As product managers, we are the people who our teams turn to in order to guide them on what to work on next. We are expected to have a convincing plan, a compelling strategy and vision for our product or our domain. But what happens when you have no clue (or low confidence) about what to do next?

Wether you’re at an early stage startup or a more mature company, uncertainty is a constant. You might be in a pre-PMF environment where uncertainty is second nature, or you could be launching a new domain or product in a mature company and face the same challenge.

When this happened to me (and it has happened a few times) I turned to the main mantra of product management:

“Turn to your users to find the answer.”

I started researching the market, interviewing users, and consulting with seasoned industry professionals. Sometimes it worked, but other times it didn’t provide a clear direction.

The last time I faced this challenge was recently, when I was asked to launch a new domain at my current company. Despite having a high-level idea of what we wanted to do, we were uncertain about the approach and strategy. We were definitely not sure if the whole initiative would bring in the results we were hoping for.

At first, I started researching and researching - market, data, users. While I occasionally felt I was onto something, I still lacked confidence in where to begin, let alone a more long term strategy. I realised that deeper I dug, the more lost I felt. So, I decided to take another approach.

I stopped researching, accepted uncertainty and put together a document outlining the opportunities I wanted to pursue. I set my objectives and key results to evaluate success. You might wonder “but if you were so uncertain, how did you manage to create something that made sense?” The answer is simple. I didn’t. I just knew that I had to start somewhere and hoped that action would bring knowledge.

It did, but the road was not paved with roses. A few weeks after we started actually working on the opportunities laid down, we struggled to see how they would fit into the bigger picture and the objectives we had in place. At that point, it was apparent that we had to significantly change our approach and discard our OKRs. The cost was about 3 weeks of work, which, however, were not completely in vain. We still managed to release features that provided some value to our users, even if we didn’t get to fully reap the fruits of our efforts.

On the bright side, we learned that our initial approach was wrong and understood why, which is a valuable insight. Then, we were in a position where we could decide on a new approach and this time we didn’t set any clear OKRs. We had a small hint on what would probably be the right metrics to track (and we did monitor them), however we didn’t want to waste our time once again, trying to figure out the right objectives, and keep overthinking about it. We just wanted to test out our new approach based on the learnings that we had from our previous failure and validate if it’s right.

Fortunately, we hit the jackpot this time. From the first few days, we felt confident that our new approach - informed by the learnings of our previous attempt - made sense. We still focused on learning quickly and we still didn’t bother setting up any fancy KPIs and targets just yet.

The key takeaway is:

“When you don’t know what to do next, just start doing.”

It doesn’t really matter if your initial thoughts about the problem are right or wrong on. Put your best effort into creating a plan with some reasoning, even if it’s based only on beliefs and convictions. By trying things out and being open to learn, taking action will show you where you are wrong and then you can adjust if necessary by utilising the knowledge gained from your previous failure. In our case, we were lucky to find what appears to be the right approach in the second try. However, in the past it took me way more effort to get things right. All you need is the appetite to learn and having the openness to being wrong. It might cost you a few weeks, but it’s way cheaper to fail by doing, than getting burned out into never-ending research.

One of the main tools we use to create alignment as product managers is documentation. As my manager often says:

“As PMs, writing is our interface with the outside world. We need to pay careful attention when crafting our documents, because sharing a bad doc is just as damaging as releasing a bad product.”

A couple of weeks ago, I had an experience that reminded me it’s not just about creating a great document, but also knowing whom to share it with. It might sound basic, but sometimes we need a refresher on the fundamentals.

Let me share the story.

We were working on a major project that was our primary focus for the quarter. After several days of effort, I finalized the relevant PRD (Product Requirements Document). I was proud of this doc because of the detailed work I had put into it, believing it would help the team and stakeholders clearly understand the problem and align on the strategy and solutions.

After completing the document, I posted it in the relevant Slack channel for initial feedback, tagging all the key stakeholders, including a highly influential one who was closely following the project.

A few minutes later, this influential stakeholder asked me:

“What are PRDs and why do we need them?”

I didn’t panic. Instead, I thought:

“Great, this is an opportunity to explain the tools we use and promote product thinking within the organization.”

I wrote a short explanation about the purpose of PRDs, confident that this would settle the matter and we could move forward. But then I received another notification:

“We don’t need such docs. They’re all fluff. Just create a doc with five bullet points highlighting the main ideas. That will be enough for the team. If further alignment is necessary, we can discuss it in a meeting.”

My initial reaction was frustration. “How could someone think that?” I wondered. But after taking a step back, I realized the problem wasn’t the document itself, but my decision to share it with the wrong audience.

No matter how well-written or thoroughly researched your document is, if you share it with the wrong audience, it won’t matter. You need to understand what kind of information each reader needs, their focus, and the appropriate level of detail for each, based on their perspective.

It seems obvious, but we often forget this. In that situation, I thought, “Why should I create different versions of the document? It’s just duplication of work.” But the reality is people rarely filter out information themselves, so it’s our responsibility as facilitators of alignment discussions and creators of documents to tailor the level of detail and focus to what is relevant and useful for each reader.

A classic example highlights this difference: for the C-suite, you’ll want to create a more abstract document, focusing on a high-level overview and business results rather than specific solutions. For the product team, you need detailed information about the problems and solutions to ensure alignment within the team delivering them.

The structure and content of your documents should reflect each audience’s different needs.

Most product people have found ourselves in a position of having a streak of what appears to be unsuccessful experiments. We are then often inclined to think that the hypotheses we used as the basis for those experiments were wrong. But were they really? Of course, it’s expected that a percentage of our hypotheses will not be valid, however, it’s also likely that the way we define success or even the segments we select (or don’t) to target could be the drivers of this failure. In this post, I’m sharing a few thoughts on this matter.

Defining success

Since every product is serving its customers in order to achieve a business goal, as product people we - naturally - often obsess with the metrics or actions that are tightly connected with driving revenue growth. As a result, our first instinct while framing a hypothesis is that the cause and effect relationship will instantly influence revenue or an action around it. For instance, we’ll usually say “by allowing my customers do X, they will buy more” or “by removing this obstacle, I’ll increase subscriptions”. However, it’s quite rare that a single action will have such a direct cause and effect relationship with revenue generation events.

In every business, there are input and lagging metrics. And, unfortunately revenue and revenue generating actions usually belong in the latter category. This means that it’s very hard to see a an immediate influence of a single action on them. However, while framing a hypothesis, we very often imply a direct relationship between the cause (by doing X) and effect (I can increase sales). So, a common mistake that we make, is that we define the success of our experiment by tying them into a lagging indicator such as sales, subscriptions or revenue. As a result, we might end up mistakenly considering an experiment failed.

The obvious way to avoid this is by defining the success of an experiment based on an input metric that it’s more likely that we can influence. But this is still not as easy as it sounds. A basic pre-requisite for that is to have a deep understanding of the levers (input metrics) that move the treadmill (lagging metrics such as sales or revenue) and their cause and effect relationship. For instance, for a streaming service, the hours streamed (input metrics) could influence retention (lagging metric).

As much as a no-brainer as that sounds, I’ve come to realise that a vast amount of product teams don’t have a great understanding of the levers that move their lagging metrics. And don’t get me wrong, it’s not an easy piece of knowledge to acquire. It takes months or years of experience on a domain, lots of contact with the users to understand their needs how they operate and even more experimentation in order to understand cause and effect relationships between actions and results.

Of course, having the way in which we measure success is not necessarily the only factor that could mislead us in defining an experiment as failed. The other thing I want to talk about is this post is segmentation.

Selecting the right segment(s)

In cases of products with large (in the scale of millions) and diverse user bases, you may often observe a perceived inability to substantially influence metrics. A common pitfall in this case is the selection of a very broad user segment for the tested hypothesis.

There are many cases where we might get carried away with forming the right hypothesis, or by thinking of the right solution to validate it and we ignore one of the most important things: Thinking of the right user segments that this hypothesis concerns. Next thing we know, is that we launch an experiment in a huge and diverse pool of users and a couple of weeks later we see inconclusive or misleading results.

What we might realise though if we look closer is that this is not true. If we look into more granular segments of our user base, we might realize that our hypothesis was right and we managed to achieve the result that we wanted. We just managed to be successful on a smaller fraction of our users. In general, the larger and more diverse the denominator - the number and most importantly the segments of users - that we use to evaluate the success of our experiment on, the more information can be lost.

So, next time we frame a hypothesis, one of the things that we should pay more attention is to clearly define the segment(s) that we believe that our hypothesis is addressed to. Ideally, we should also make those segments as specific as possible, while of course keeping in mind that they should still have a substantial size in terms of volume that would still allow us to run an experiment and get statistically significant results.

Hopefully, the thoughts expressed above can not only help you get a better success rate for your experiments, but also gain more insights about your users. At the end of the day, we don’t just run experiments to drive better business results. Knowledge acquisition for our users is also vital reason for running them.

Are there any other things that you’ve observed as common pitfalls while running product experiments? I’d love to learn about your own experience.

A couple of weeks ago, we had a chat with a few PM colleagues at the office regarding PM’s involvement in delivery. The subject of the discussion was the common struggle where a PM gets constantly involved in delivery way more than they’d like to. The (frustrating) consequence of that is that this often eats up so much of their time, that there’s no time left for other important tasks, like discovery.

In my experience, this problem is very common across companies, so I started to ask myself: Is there a way to solve this? So, I started trying to first figure out the reasons behind this and a few things that could help.

Are we on the same page regarding business objectives and ownership?

My first thought was that as product managers, we have a natural predisposition to - at least be concerned about delivery - which is sourced from our ownership. PMs are naturally considered to be the main owners of business objectives and, inevitably, business objectives rely on deliverables. This, by itself, creates a tendency to be involved in delivery, which frequently ends up with the PM managing it, as falling behind in delivery, usually means that you’re falling behind on your goals.

However - ruling out the case where a PM is a micromanager - delivery is one of the duties of the tech lead. Consequently, shouldn’t they also have the same ownership in business objectives as the product manager?

So, this was one of the first issues that I looked into. Does everyone on the team - and especially the product trio - share the same sense of ownership of business objectives? Hopefully yes, but being honest, I have seen (and I’m sure you have as well) cases where TLs only care about engineering excellence and how the tech stack looks like, design leads only caring about UX best practices and PMs about writing good stories and creating fancy roadmaps, instead of delivering actual business value.

Are we putting too much on people’s plates?

During the aforementioned discussion - a colleague brought up an argument that stuck with me - and truth is many PMs frequently ignore:

“Often, we ask from the tech lead to manage people, write code and contribute to deliverables, manage delivery, participate in shaping and other ceremonies, and occasionally address business queries as well. We often ignore that “they only have two hands” and can’t do it all. Inevitably, when delivery falls behind, PMs are going to step in to manage delivery. And then the domino effect starts, as they’ll also have too much on their plate and areas like discovery will fall behind.”

The point I am trying to make here is that as PMs, we often focus on the multitude of tasks that we have on our plate due to the versatility of the role and forget that this might be happening with other people as well. The above example focuses on how you can overload a tech lead, but this can be the case for anyone on the product trio. For instance, a designer might be forced to spend way too much time on being involved in research or framing and so on.

Taking a step back, the main consideration here is: Are our roles well defined within a team? And if so, do we protect the boundaries of our roles, making sure that people actually have the capacity to do what they’re asked for? Does the company protect those teams by giving them the necessary resources, so that PMs won’t have to fill in gaps in delivery management, TLs won’t have to fill in gaps in product development and designers won’t have to fill in gaps in product discovery?

Is the team empowered? (not in the “fluffy” way)

With the term “empowered” here, I don’t want to go to the “fluffy” definition about “empowered product teams” which work on a pink cloud and do everything perfectly. I mean can the team make decisions while delivering without the PM signing off everything? Or are you the bottleneck?

Unfortunately, there are many cases where the PM themselves are the bottlenecks when it comes to delivery. And while you are not “actively” managing delivery, you affect it in a significant way. There are cases where nothing finds its way to delivery, unless the PM signs it off. That means that the delivery team cannot make decisions by themselves and as a PM you will have to always give the green light. There could be several reasons for that, to name a few:

1. The team might not feel that they have the necessary context to make a decision. And this is a PM problem, because if that’s true, that means that the team does not understand well what they are doing and what we are trying to achieve. And this is a product manager’s problem to fix, by improving how they bring and communicate projects with the team.

2. The team might simply not have the permission: Again, a PM’s problem to solve. If a PM is not allowing the team to make decisions on their own, that ultimately means that they don’t trust them.

The bottom line here is that the PM is not “the boss” of the developers or the designers and they need to give them the necessary trust and context so that they can make decisions by themselves.

Let’s jump off the pink cloud now

Having considered the above, I must be realistic. We don’t live in a world created by Marty Cagan. Business is business and there will always be lows. There will always be times when there will be pressure as you are falling behind from your goals, or a new time-sensitive opportunity presents itself and you will have to be more tactical. That would mean that short-term and delivery is a much higher priority and areas like discovery will matter less. In that case, you will naturally gravitate more towards delivery and it’s ok.

The thing here is how often will that happen? It’s not uncommon for companies to be in that state of emergency constantly and the focus on execution to be the only thing that matters for a long time. And this can become the reality of the company, in which they might forget that the strategic matters as well. Being able to think more long-term, discover new opportunities and find ways to nurture them can only be on the back-seat for a limited amount of time.

Do you agree with the above perspectives on why product managers are spending more time than they should on delivery? Have you witnessed any other reasons why this happens? If so, I’d be glad to listen to them. Hit reply and let’s talk.

During my summer holidays, I trained over 150 AI models. Why? Firstly, I find working on something new and creative very fascinating, but most importantly, as a product manager, I have never worked on an AI-focused product. So, I wanted to gain first-hand experience working on such a project and possibly gain some learnings that could benefit me professionally.

With that in mind, I looked into several public datasets and decided to work on one where I would classify patients as diabetic or not based on eight different parameters from their medical history. The best result I achieved was a 96.3% accuracy on my model predictions (using a separate validation dataset). Although accuracy is not necessarily always the most important evaluation metric for an AI model, in this particular case, I only relied on this to evaluate my models for simplicity.

In this post, I will go over the story and my learnings. At the end, you can find a link to my GitHub repository with the code I used to develop and train my models. You can read the full story first and then proceed with the learnings or skip directly to the learnings section. It’s up to you.

The story

Preliminary Decisions

As soon as I decided what dataset I was going to work on, I had to make high-level decisions about my approach, as this would dictate some of the work I would have to do later on. In this case, I was dealing with a simple binary classification problem, where I would have to classify patients into two classes, diabetic or not. So, the first high-level decision I made was to tackle this problem using a neural network, which is the designated (not the sole though) tool for classification problems. This was the first high-level decision of importance, as it dictated the libraries, tools, and frameworks I would be using later on, as well as the preparation work I would have to do on my dataset. For instance, I knew I would use tools such as Numpy to work with my data and Tensorflow to train my models.

Working with my dataset

Accordingly, I knew I had to prepare my data to be ingested in a neural network. First and foremost, that meant that in its final form, my dataset would need to have a specific format and I would need to split it into three segments:

• The training dataset that would be used to train my models consisted of approximately 70% of my original dataset (100,000 total samples).

• The test dataset that would be used during training to evaluate the accuracy of my models, and based on those, I would pick the best model (15% of the dataset).

• The validation dataset that would be used on the selected best-performing model, to verify how the model is working in the wild (15% of the dataset).

Each of those datasets would be split into two vectors. The one with the training params (the X params) and the labels that indicate the final result (the y params).

However, this would only be the final form of my data. Before that, there was a long journey that would be split into two distinct phases: In the first one, I was exploring my dataset, and in the second, I had to get my hands dirty and prepare my data.

Exploring the dataset

If I would say just one thing about this phase is that I never imagined how important it would be. Before I started, I naively thought that the structure and content of my data would be straightforward and largely expected. I should be smarter than that after working ten years in software, knowing that many things could go wrong in a large dataset. You can see unexpected values, and you will certainly have missing values or even unexpected data types.

I accidentally realized that while looking at the labels of one of the data categories, the gender. How many distinct values could exist under this category? As you can easily imagine, more than I expected. Once I realized that, I went over every single category, and alongside many surprises, I realized that I should spend significant time preparing my data. I would also have to make some decisions on how I would use them.

For example, one of the categories was the smoking history. The values under this category were “No Info”, “current”, “ever”, “former”, “never”, “not current”. The problem was with the “No Info” value, which accounted for approximately 35,000 samples. This meant that for about 35% of my dataset, I practically had no info about their smoking history. In this case, I had to make a decision. What would I do with this category? I had some options:

1. Use the “No Info” label as a separate label and keep these samples in my dataset.

2. Get rid of the samples with the “No Info” as a label, which was an unattractive option since I would lose 35% of my dataset.

3. Get rid of the smoking history category as a whole, but keep 100% of the rest of the categories of my dataset.

At that point, I decided that I didn’t have enough knowledge to pick one of the above options, so I decided to create three variations of my dataset and later on feed them to a simple model to see which of those options would work better in terms of accuracy and double down on this one. Sneak peek; the first one worked better, so this is the option that I took. In general, there were several similar small decisions that I had to make while preparing my data.

Preparing the dataset

Once I understood my data well, which included visualizing them, I started preparing them to be ingested in my neural network. That involved small and other more extensive tasks, such as:

• Trimming unnecessary rows (e.g., headers).

• Replacing values such as strings with numbers to be easier ingested in the model (e.g., replacing “male”, “female”, and “other” with 0, 1, 2 labels).

• Converting them to data types that would enable me to work more efficiently.

In general, this phase didn’t take long. As soon as I knew what were the jobs to be done, it was pretty easy for me to do that. However, I reckon that in a large-scale project, there should be certain pipelines and processes that should be taking care of the data preparation part, as it would be integral to the success of such a project.

Thinking about the model

On a high level, what you need to understand about neural networks and how they work is the following: Neural networks are consisted of layers, and their layers consist of nodes (the neurons). The data are initially ingested in the input layer, and once the neurons process them, several parameters are extracted and then ingested into the next layer to be processed by its neurons and so on. Then, in the case of a binary classification model like the one I was working on, the parameters are ingested in the output layer, which is consisted of two neurons (because there are only two potential classes), and we get a decision on if our patient is diabetic or not.

In terms of layers and neurons, one of the decisions I had to make was the general architecture of my model. In other words, how many layers would I use, and how many nodes (neurons) each layer would have? The more layers and nodes you have, the more complicated the architecture, which means the more time and resources you need to train your models and, in some cases, the higher the risk of overfitting your model (meaning to make it perform great on your training data, but poorly out in the wild).

My initial idea was to create three different model architectures, a simple one, one a bit more complicated, and a super-complicated one, and see which one would perform better. Then, based on the result, double down on this kind of architecture to further optimize it. I quickly realized that this idea wouldn’t get me too far. To begin with, the results were at best inconclusive. There were only slight differences in the performance of each architecture, although the simpler ones gave a small indication that they could be performing better.

Still, I realized that trying to figure out an optimal architecture like that was like shooting in the dark, trying to hit the jackpot. So, I decided to take a more experimental approach.

The optimization function

This is when I decided to use an optimization function. This function would practically create and train several models based on every potential combination of layers and nodes per layer. It would allow me to pick the model with the best performance. For instance, if I wanted to try every potential combination of 1, 2, or 3-layer architectures, and each layer could contain anywhere between three to seven nodes, there would be 155 possible architectures that I would need to try. As you can easily understand, it would take forever to do this manually. However, it only took a little less than 40 minutes for the optimization function I created to compile and train those models for me and let me pick the best one.

In particular, in those almost 40 minutes, I used a MacBook Air to train 155 models. The best result I achieved was 96.3% accuracy, while the worst was 91.2%. In this case, the amount of data, models, and their complications were reasonable, so the time and resources needed to train the models were well within grasp. So, I didn’t need to do any resource performance-oriented optimizations. However, this would be essential in larger-scale projects, or else the cost can easily get out of hand very quickly.

That was it. In terms of duration, the whole project took me about eight days from start to finish. Each day I worked on it for an average of 2.5 hours.

If you are interested, you can find and download the final form of the code I used here, run it yourself on your machine, and see the results.

The Learnings

In the following section, I share the main thoughts and insights derived from this experience. While thinking about them, I approached the issue by asking, “what do I keep from this experience if I wanted to apply my learnings on a similar - larger scale project (e.g., at work)? Well, let’s see:

• Data is king. As fancy as AI and the models sound, the data is the most valuable asset and the make-it-or-break-it part. If you don’t have the data that you need in terms of substance or if they are not sufficient, no matter how sophisticated the models that you are using are, the project is doomed to fail.

• Data collection, storage, and processing must become a core part of the product culture to succeed.

• Data fluency is a must. Having people with experience in working with data at scale is essential. They could be data scientists or data engineers, however, I cannot imagine working on a large-scale project and this skillset to be missing from a product team. Several decisions need to be made with an impact on your models’ performance and results, but also on the resources and cost.

• As an AI product manager, a big part of your job will be to have a good understanding of the feasibility of a project in combination with its value. Several parameters can inform this decision. Broadly they are summarized in questions of technical and business nature as well. In particular, here are some of the questions that you might need to answer:

  • Can the AI system that you design meet specific performance requirements? (Technical)

  • How much data are necessary? (Technical)

  • What’s the engineering timeline? (Technical)

  • Will that lower the costs? (Business)

  • Could this help us increase revenues? (Business)

  • Will this model help us launch a new product / business? (Business)

• You also need to have a deep understanding of the problem you’re solving to pick the proper evaluation and acceptance criteria for your models. Is it accuracy? Are you tolerant of false positives? Is it recall or something else? For instance, if you are developing a model that is used to predict the likelihood of a patient getting cancer, you might not want to take any chances of a mistake, so you might be more tolerant of false positives so that you will eradicate the possibility of not diagnosing a patient correctly.

• The training and model optimization processes can be extremely resource heavy. So, you need to be prepared accordingly regarding the budget or find other smart ways to avoid some of the costs. Several pre-trained models out there can give you a head start and help you reduce your costs, or other tools such as Google Colab could help.

• There’s no silver bullet when it comes to model architecture. You need to have an experimental approach, try out several iterations, and this cannot be a manual process. This could take more time and cost more, however it will increase your chances of developing the best-performing model that you can. However, more often than not, the more you complicate things, the worst results that you will get.

In conclusion, it was a super-valuable experience for me, bringing me a step closer to better understanding the world of data and AI. Hopefully, some of the above learnings will be useful in practice for you as well, however, if you’re like me and want to familiarize yourself a bit more by getting your hands dirty, I strongly recommend doing something similar.

A few weeks ago, I had a coffee with the product director of an eCommerce SaaS company. At some point during the discussion, he brought up that their NPS was at 10 at the time and that one of the annual targets of the product team was to get this number to at least 25.

Now, let’s go a few years further back in time. When I first heard about NPS as a metric, I thought it was a very interesting concept. The whole narrative of promoters and a self-sustained growth machine sounds very attractive and easy to grasp, especially if - like any product person that gives a damn - you want to see your user count grow. Soon enough, I realized that NPS is significantly flawed as a metric when tracked in the context of a product team and is used as a KPI based on which significant decisions are made.

Returning to the discussion from the beginning of the post, I started wondering why they emphasized the NPS so much and believed it to be so valuable that it ended up being used as a goal for the product team. So, I asked to see a small sample of their NPS responses to see what they were seeing for myself. And this actually verified my aforementioned opinion about the flawed nature of the NPS.

Still, NPS is a quite popular metric among product leaders and product teams. In a LinkedIn poll, I ran a couple of weeks ago, at least 4 out of 10 product leaders said they are tracking NPS in their product teams and making important product decisions based on this score.

In this article, I want to showcase why I believe this is not a good idea and that product leaders should look into other metrics with more substance to make product decisions.

The conception of NPS

Net Promoter Score was invented in 2003 by Fred Reicheld as a simple way to measure customer loyalty that could be applied across various industries. It’s calculated based on the customer’s answer to a single question: “On a scale of 0 to 10, how likely are you to recommend [company] to a friend or colleague?”

Then, you simply get the percentage of people who responded with anywhere between 0 and 6 (the detractors), and you detract it from the percentage of those that responded with 9 or 10 (the promoters), and there you go. You have your NPS score that can vary from -100 to 100. Simply put, the higher this score, the better for the company. But people also advocate that you will likely create “a self-sustained growth machine” when you get many people so happy that they fall under the promoters category since they will recommend you to their friends or colleagues.

So, before we jump into the main flaws, let’s start with the apparent. NPS was not invented as a product metric but as one intended to measure loyalty. It could potentially be of value for teams that are specifically working on product growth, but it’s far from being a metric of general use. But let’s see why else I consider NPS to be flawed.

One question to rule them all? Why?

The “one question to rule them all” rationale seems quite questionable to me. In some contexts, this question (how likely are you to recommend [company] to a friend or colleague?) can make sense. Mostly in B2C contexts. However, in most B2B contexts, not so much. Many respondents of NPS surveys run by B2B products also agree with that view. Specifically, in the sample of responses that I talked about at the beginning of this post, there were quite a few cases with answers from 6 or below (detractors), who left a comment along the lines of “why should I recommend a software for work to a friend or colleague?” So, suppose that you were the product manager of that product, and you saw those responses. What would you make out of them? Would you trust them? Is this person really a detractor or just someone confused because of an out-of-context question they were asked?

Also, even if you actually want to measure customer loyalty, why should all companies and industries be using the same way to measure this? Each product and its business has its own characteristics. As product people, we should be able to measure loyalty or organic growth more accurately. For instance, from its very nature, the NPS question resembles a question asking for feedback on an intention of referral. So, you could launch a referral program, and instead of asking people to answer the NPS question, address the people you know that are actually neutrals or detractors (i.e., those who haven’t referred someone) and ask them what prevents them from doing so. The feedback you will collect this way will be way more accurate, and most importantly, you will be way more confident that you are asking the right people.

The above is just an example. I am not proposing that this is how we should measure loyalty or get feedback from customers who do not belong to the “promoters” segment. But I suggest that each product team should think more deeply about the best way to measure customer happiness and success based on its unique traits.

People have different evaluation criteria, and responses in scalar measurements are not directly comparable with each other

When people are asked to rate something on a scale, as they are asked in the case of NPS, their responses are not directly comparable. This happens for a single reason: If I evaluate a product or service with a 7/10 and another person does the same, that doesn’t mean we are equally happy. For example, one of us might be extremely satisfied and a frugal rater at the same time, while the other one could be dissatisfied but still a generous rater or a people pleaser.

Getting back to the sample of NPS responses we were talking about, I ran an analysis focusing only on the responses of the detractors. Out of 128 of those responses, I could confidently claim that at least 35% of them were not actual detractors. They were either neutrals or promoters, and they just were frugal raters. How do I know? From their comments in their responses. They all looked something like the following: “Amazing,” “best tool I ever used,” and “happy so far.” So, what do you think? Are those people really detractors? And most importantly, would you really rely on a metric to make crucial decisions when 35% of the responses of unhappy customers (11% of total responses) are clearly inaccurate or untrue?

Influenced by many different factors

Another critical aspect to consider is that NPS responses can be affected by many factors, irrelevant to the product per se. The question used to measure NPS is quite general so that a response could encapsulate the customer’s perspective for any area, from the product itself to customer support or their experience with their sales team. A good or a bad response could be connected with anything. So, as a product team, you are tracking a metric that could encapsulate aspects far from your influence.

To make it more accurate, getting back to the example I used before, several answers from detractors were complaining about customer support. In particular, people claimed it occasionally took too long to respond to or fully resolve their issues. In this particular case, as a product team, you are optimizing based on a metric skewed by a different team’s efforts.

Not specific enough - ends up being a distraction

To build more on the previous argument, NPS doesn’t give you specific information about what’s going wrong or what you are doing right. Consequently, it is not an actionable point of feedback. In the sample I came across, at least 27% of all the responses were not specific. They either entirely lacked a comment (the majority of them), or even if they had one, the comment was too generic. It could be something like “buggy,” “not good enough,” or “needs improvement.” Yes, this could provide you with a sentiment of the user about your product. Still, it’s primarily a distraction from meaningful feedback and insights collection efforts, such as user interviews.

It can make you lazy

I have often encountered cases where it’s hard to directly connect a feature with a measure for its success. So, it’s not uncommon to answer the question “how do we know if we’re successful or not” with “we’ll measure the NPS of people that used that feature.” Of course, this has absolutely no rational grounds as an approach and it’s just lazy. When shipping something, you need to have a clear metric that shows you if you’re successful or not and need to be able to connect this metric with value to the user. In such cases, choosing just to measure NPS is only a shortcut that prevents you from thinking hard about how a feature would add value to a user and how you can measure it.

Wrapping up

So, that was my honest (and probably harsh) criticism on NPS. While it’s a quite popular metric and when it’s high enough, it makes management, investors and even teams happy, it cannot be used as a measure of success for a product. If you do use it though, do that with caution and even when using insights from NPS surveys as an input for product decisions, make sure to inform those decisions with further and proper research with users and customers.