ISSUE #6 - JUNE 8, 2022

5 things to watch out for when reading product metrics

Reading reliable metrics and interpreting them properly is the cornerstone of making the right decisions on where and what to work next. Here are five fundamental tips that will help you ensure you are on the right track.

5-thinks-to-watch-out-when-reading-product-metrics-1-2

A significant amount of a product manager’s time is dedicated to interpreting how the product performs, aiming to identify areas for optimization. This heavily involves reading metrics and KPIs. As explained in my previous post, tracking the right metrics and KPIs is vital in our effort to ensure that we are working on the right problems. However, this is only the first step. Their reliability and the way we interpret them are the second part of this equation.

While “reading metrics” sounds straightforward, it is a bit more complicated than most people realise. For example, let’s assume that as product managers, we are working on an online consumer product. Our users pass through a simple funnel of a few steps to complete a purchase. This is the main goal of our product. In that context, one of the metrics we are trying to optimize is the conversion rate (purchases to visits).

If our average conversion rate in a given period of time, let’s say a month, is 4%, what conclusions can we really reach based on this information? Does this information help us understand if our product is performing well or not? Does this information allow us to understand where we need to work, so that we further improve our product’s performance?

Not really. If this is the only information at our disposal, the only thing that we can say with certainty is the following: For this given month 4% of the visits to our website ended up in a purchase.

Yet, if we dig deeper, we can reach some more helpful conclusions. So, let's go over a few of the fundamentals that we should be watching out for while reading our product metrics.

Ensure that the metrics that you are looking at are significant

A percentage alone without a volume is useless. It is very different to be looking at a percentage that derives out of a sample of 100,000 data points, than one of 100 data points. The insights that we can take from the first case are much more significant and reliable, compared to those of the second case. In small samples, even the slightest of changes can produce significant differences in the final results.

On the example that was mentioned earlier, the main problem with this 4% is that we do not know how significant it is. Is it out of a sample of 100 users or 100,000 users? As you understand, in the first case, the number that we are looking at has no significance at all. If we had a 4% conversion rate, with data points from 100 visits on our website, that means that we have only 4 conversions. This is not a significant number, nor a reliable one. A tiny change of even one more conversion would take our conversion rate to 5%. Compared to the 4% that we measured initially, this is a 25% increase.

In the case where this number was a result of 100,000 data points, that would mean that we had 4000 conversions. Now, the same increase of only one conversion would have a very negligible impact on our conversion rate. It would become 4.001%, which is only a 0.02% increase. It's easy to conceive that the bigger the volume of our sample, the more significant and reliable the numbers we are looking at.

The world doesn’t work in a straight line

Often, we are talking about average numbers that refer to relatively extended periods. In our example above, we have a 4% average conversion rate that corresponds to a month. The problem here is that this number is a product of many fluctuations. Many peaks and valleys have occurred, depending on the smaller time increments that we would observe. For example, maybe during the first week of this month, the same metric was 6%, whereas in the second it was 3%.

When someone says that a funnel’s conversion rate for a given month is 4%, people usually imagine a straight line. That would mean that week over week, or day over day, for this month, people actually convert with a consistent 4%. This is rarely true. In the vast majority of the cases, averages are a result of fluctuations that look like this:

perception-vs-reality-on-reading-metrics-

Understanding those fluctuations is essential so that you can cut out the noise. Those fluctuations could be triggered by various things. For instance, they could be a result of seasonality, which is quite common in consumer products. There could be macro-economic factors affecting our users’ behavior, such as bad economy. Especially when observing bigger time frames. . However, it could also be something related to the product itself. Like a change we recently shipped, or another issue that is preventing our clients from buying. It is our job as product managers to be able to understand the root causes of those fluctuations so that we can act whenever needed.

Those root causes are highly relevant to the specifics of each product or of the industry. As a product manager, you must familiarize yourself with those specifics and get a good grasp of them. That will let you distinguish between random fluctuations and a potentially significant problem with your product.

Compare with the right benchmarks

A data point is nothing, if we don’t have one more (or plenty more actually) to compare it with. Benchmarking is tricky though. Comparing with the right benchmark is essential so that we can make the right decisions.

Getting back to our example: assuming that our average conversion rate is 4% in a given month, out of a sample of 100,000 users, can we tell if this number is good or not? No, if we don’t have another data point, we can’t. But which one should this other data point be?

This is a quite tricky question and a head-scratcher for many product managers. There are quite a few heuristics in use, trying to answer this question. The goal here is to make sure that you are comparing apples with apples.

For example, you could compare with the corresponding week of the previous year. The problem here is that you are comparing with a period with a probably different macro-environment for your users. Not to mention that your own company and product have likely changed a lot. So, by definition (unless things have gone terribly wrong) you should have improved. On the other hand, comparing with the exact previous month could be a bit more reliable. But still, you would not be comparing apples with apples exactly.

A classic example is the case of eCommerce. In eCommerce, usually, the middle weeks of the month are not that good. During those weeks people don't have enough money to spend. This is actually depicted in the conversion rate of those businesses. On the contrary, the last and first weeks of each month, are usually way better in terms of sales (or conversions). People are getting their paychecks and actually have money to spend. So, imagine a scenario where you are comparing the third week of a month with the fourth one. It is highly likely that the fourth week would have a much better conversion rate. But this improvement is only seasonal and superficial.

This gets us back to what was mentioned earlier. We must know the fundamentals of our industry. Knowing how your industry works will help you understand what is the right benchmark to compare with.

Avoid Aggregates

Aggregates are great in giving us a quick and general picture of where we stand. The problem with them is that they are hiding precious information underneath. This information is of much higher quality and could show us where problems lie in our product. As a result, we have the right information and can prioritize the right items into our roadmap.

What if our traffic consists of 70% mobile users and 30% desktop users? Wouldn’t it be important to know if there is a performance difference between those two segments? For example, desktop may be performing better than mobile. That would mean that there is some good space for optimization in mobile. Especially, since mobile accounts for the vast majority of our traffic. Case in point, different segments are highly likely to have different performance.

Breaking down our numbers into segments is a great tool that helps us identify opportunities for optimization. As a next step, we need to do further research to understand what the exact problem is. That's the hard part. Then, comes the fun part of thinking of potential solutions.

A product manager should be able to identify which are those segments that matter. Depending on the nature of the product, those segments could be anything. From user demographics and traffic sources to more technical segments such as device types, operating systems, screen sizes, or browsers. And of course potential combinations of those. For instance, a product with users in many countries could have significant performance differences in different user demographics. Or even within the same country, people using different operating systems, devices, or screen sizes, could have a significantly different experience while using a product. This will often be depicted in the product metrics.

Ensure your metrics are not skewed

Last, your sample should be as consistent as possible and not skewed by external factors. Imagine that from one week to another, you observe a 25% drop in your conversion rate. You start an investigation in order to identify what has gone wrong. Then, you realize that over the last week there is a huge increase in traffic. The source of that is a new campaign that your marketing team just launched. Furthermore, you find out that this campaign brought in many window shoppers, people with no intention to transact. At that point, you realize that the performance drop is not really an issue of the product itself. You are dealing with a false alarm.

Anomalies like the above could be sourced from anywhere. Unfortunately, there's not a standard formula to help you pinpoint them. Each product has its own methods and heuristics on that. A classic case is traffic which is sourced from campaigns. Especially those that are in experimental stages. Traffic from such a source is quite likely that has inconsistent behavior. That means that limiting the sample to users whose behavior is more predictable is the way to go. That for example could be the case for the users that land organically on your website.

Concluding

The quality significance and the reliability of those metrics are the fundamentals that help us as product managers understand how our product works. Reading them right, helps us make educated decisions on where we should focus and dig deeper in order to further understand the underlying problems and think of the proper solutions. This is essential so that, alongside our team, we contribute to the continuous growth of the organization.

Past Newsletters

Browse All

ISSUE #5 - MAY 5, 2022

Selecting product metrics & KPIs

How can we select the right metrics & KPIs for our product? In this post, we apply a simple framework to a case study and select primary and secondary..

Read here