ISSUE #5 - MAY 11, 2022
Selecting product metrics & KPIs
How can we select the right metrics & KPIs for our product? In this post, we apply a simple framework to a case study and select primary and secondary metrics to help us track our product performance?
Tracking the right metrics and KPIs has been a challenge for many product organizations and rightfully so. Unless you are tracking the right metrics, you cannot maximize your chances of success, as it would be impossible to ensure that people are working on the right problems. Furthermore, teams would not be able to understand the impact of their efforts.
In particular, tracking the right metrics is even more challenging in the case of product teams. Products are usually big and complicated, often consisting of multiple teams that are contributing to certain areas. As a result, selecting the metrics that are meaningful and insightful at the same time is a complicated process. To make things harder, each product is different, so there is no one-size-fits-all solution when it comes to product metrics. The good news is that there are frameworks that can help us select the right metrics and KPIs. In doing so, product organizations can track the performance of their products, as well as the impact of their team efforts.
The purpose of this post is to showcase such a framework. In order to do so, we are going to use a simplified case study. In this case study, we are going to try and identify the metrics and KPIs that we would track if we would be working for an online travel agency. To clarify, the list of metrics we are going to come up with is by no means a comprehensive list of metrics to track in general, but a set of metrics that would make sense for this particular case that we are exploring. The important part that you should take out of this case study is the process that we will follow in order to come up with this list.
The Framework
Firstly, let's briefly go over the framework we are going to use. Starting from our business goals, we are going to follow a top-down approach and try to understand how those goals are translated into value for the users and how this value can be measured. We will go over the following steps:
- Understand how our product is generating revenue for our business
- Connect revenue generation with value delivered to the user
- Understand how product areas affect the value delivered to the user
- Identify metrics that help us measure the performance of those product areas
The above framework will be applied on two levels. The macro-level and the micro-level. The former concerns the product as a whole, in other words, is more connected to the big picture. The latter concerns more specific product components and is usually tied with smaller areas of the product. We will see how those two levels work in detail in the next parts of this post.
The case study
We are going to apply the above framework to the case of a hypothetical online travel agent. So, let’s try and understand on a high level what our product does. The problem we are trying to solve is to allow users who wish to travel to a destination, to compare among different options and select the best one that is suited to their needs. In order to simplify the case, let’s assume that we are only focusing on the flight selection part of the trip. So, practically, to help you better understand the case, we would be a service similar to Kiwi or Skyscanner.
Now, what most of us see when using such a service is the “front-office” part, which is usually a web app, with a few simple steps. Usually, we select a destination and our flight dates, then we are presented with a few available flight options. Then, once we select the option that we prefer, we add a few more details and then proceed to checkout. The simplified version of this flow is depicted in the image below:
However, as far as the whole product is concerned, this is only the tip of the iceberg. Meaning that this flow is only a tiny part of the product. Below that tip, there are many other product components, usually built by different product teams, that work in parallel. One team could be working on customer support, integrations with flight companies or flight aggregators, payments, operations (e.g. cancellations), or the post-sales experience.
As mentioned earlier, in order to talk about metrics and KPIs, we need to work on two different levels. The macro-level and the micro-level:
Working on the macro-level
As mentioned above, when working on the macro level, we need those metrics that tell us how well or bad the product is doing as a whole. The term that we often meet when talking about those metrics is the “northstar metrics”. Those are the most important ones, in the sense that they tell you how well or bad you are doing with a glance.
So, in our case, what would be those metrics? Let’s follow a top-down approach, leave the metrics aside for a moment, and think: How is our business making revenue? What is the core value that we offer to our users, for which they are paying us?
In the case of our travel agency, people are paying to book their flight tickets. This means that they are looking for the best possible flight choices. Depending on the audience that we address, that could be the cheapest flight, the best in terms of flight times, or the most credible flight company.
But ultimately, when people visit our website, the magic moment for us is when they hit that “Pay” button at the last step of the flow we saw above. Obviously, for each business revenue is king and it is the number one metric each CEO sees every morning. In our case, in order to generate revenue, people must firstly visit our website (that is the job of the marketing team usually), find the best flight for them, and finally checkout. So, a great indicator of how efficiently we are monetizing the visitors of our website is the ratio of purchases to visitors. In other words, what most people call, our conversion rate.
But is that all? Could we say that by monitoring our conversion rate - our northstar product metric - we know everything we need to know? Not really.
In this case, the conversion rate is our primary metric. Primary metrics are almost always affected by others, those that we often call “secondary metrics”. And this will take us to the next chapter, our micro-level.
Working on the micro-level
In spite of their title as “secondary”, this does not mean that those metrics are less important. As pointed out earlier, conversion rate, which is the primary metric in our case, is affected by other metrics. For visitors to buy - that is to “convert” - they first need to pass through multiple steps of our funnel, which directly affect the overall conversion rate.
In the case of the online travel agency that we are exploring, the user would have to go through four steps, before completing a purchase and there are various reasons for which they could drop in each of those steps. In order to go deeper into the secondary metrics, let’s assume that we are working for the product team responsible for the comparison page (the second step of the picture above) and think about how we could select those metrics that show us how well we are doing and how conversion rate is affected.
The comparison page is the step where our users see and compare various flight options. In order to define the metrics that would showcase our product’s performance, we would follow a similar method as we did at the macro-level.
So, let’s break this down. How can we deliver the best possible value to our users on this page? Users at this stage have an intention of booking a flight. They need to see as many options as possible that match their search criteria, find the ideal one for them and proceed to the next step. This means that our primary goal here is that users actually find a flight that they like and proceed to the next step. In other words, users will not abandon our page at this step.
Consequently, the metric that would give us this information is the “Abandonment rate” - the ratio that derives from the number of users that drop at this step to the number of users that landed on this step.
At this point, we identified a metric that can tell us how well or bad we are doing when it comes to this page in particular. But how is this metric affected? Given that the users are here to compare flights, that would mean for us that we should showcase a good amount of options to the user so that we ensure that they can find the one that better suits their needs. As a result one of the secondary metrics we should observe is the average number of results per user journey. For this assumption to be valid, we should observe a positive correlation between the number of results per user journey and the conversion rate. Meaning that the more results we show per user, the more likely it is for the user to convert.
But how can we affect this metric (i.e. the number of results shown per user journey)? In order to understand this, we need to know how online travel agents source the flight options that they showcase to users. Usually, this happens through API integrations with airlines or other companies which are called “flight aggregators”. Flight aggregators are companies that buy flight seats massively at wholesale prices from airlines and sell them for a profit to travel agents and other entities. Online travel agents, usually work with more than one airline or flight aggregator. So, each time online travel agents want to show flight options to a user, they do an API request real-time to the flight aggregator or the airline and those parties respond with the available flights for the flight preferences that the travel agent requested.
Those results are then shown to the user as the flight options. As you understand, since there is a request/response process going on in real-time, there are many things that might go wrong. Firstly, there could be no options available. Or there could be a bad request or unavailability of the other party. So, our goal as the product team would be to get as many successful API request responses as possible - meaning responses with valid flight options. Why? Because the more successful responses we receive, the more flight options we can show to our users.
The ratio between successful responses returned to requests initiated - let’s call it return rate - is another secondary metric directly affecting the conversion rate.
Putting everything together
So, if we summarize, as a team working on optimizing the page where users compare flight options, we are optimizing by tracking the following metrics:
- Abandonment rate: Users abandoned on this step / users dropped
- Average Number of results per user journey
- Return Rate: Successful API responses / API requests initiated
Thinking about the big picture and combining the macro-level and the micro-level that we have analyzed so far, we can say the following:
As an online travel agent, the primary metric, the northstar that we are tracking to measure our performance is the Conversion Rate. As secondary metrics, from which our northstar is directly affected, we are tracking our abandonment rate (in this case only in one step of our funnel, but every step is important), the number of results per user journey, and our return rate. So, for example, if we see our return rate dropping and at the same time, the conversion rate would be dropping as well, in a super-simplified world where no other metrics than those we described were affecting the conversion rate, we could claim that our performance dropped because of the drop of our return rate. So, the first line of action for us would be to investigate why this happened.
At this stage, we should repeat that the above analysis is oversimplified. In reality, many more factors would be affecting the user’s choice, resulting in many more metrics affecting our northstar metric. However, each product team usually follows an analysis similar to the above in order to come up with the set of metrics that will help them track the success of their efforts.
Some closing thoughts
Ensuring that we are tracking the right metrics, ensures that we identify the things that we need to improve on and we are working on the right problem. So, it is crucial that the set of metrics that we choose to track is the right one.
Similar to a product, the set of metrics that we are tracking is a living organization. What is important now, is not necessarily important in the future. The metrics that are driving success could be changing. Of course, not frequently. But with the evolution of the organization or with changes in strategy or the industry, the metrics and KPIs that show us the way could be changing as well. It is important that we can always take a step back and re-evaluate them if needed.
As a closing thought, selecting the right metrics and KPIs is (a very important) first step, however, our job does not end at that point. Reading those metrics right is as important, as selecting the right metrics. That brings other considerations to the table, such as: what is the right comparison benchmark? How do we make our observations unbiased? How do we mitigate effects like seasonality? And how do we uncover the truth hidden behind aggregate metrics? All those questions are closely connected to metrics selection. As their answers are long, this is a topic we are going to go over in my next post.
Past Newsletters
ISSUE #4 - APRIL 5, 2022
Making agile frameworks work for your team
What are the most common reasons agile frameworks do not work as expected? How can you make agile frameworks work for your team? What is the right..
ISSUE #3 - MARCH 1, 2022
Breaking into product management
How can someone become a product manager? Is a software engineering background necessary? What are the fundamental skills that make a great..
ISSUE #2 - FEBRUARY 2, 2022
The challenge of delegating
Advice for managers on when, what and how to delegate effectively. Avoid frustration, subpar deliverables and missed deadlines.
The Product Notebook by Manos Kyr.
Once every month, I’m sharing my thoughts on product, growth & entrepreneurship.
Copyright © Manos Kyriakakis