ISSUE #23 - MARCH 29, 2024
Thoughts on product experiments to drive better and more insightful outcomes
In this post, I'm sharing my thoughts on how to improve the success rate of our experiments and learn more about our users.
Most product people have found ourselves in a position of having a streak of what appears to be unsuccessful experiments. We are then often inclined to think that the hypotheses we used as the basis for those experiments were wrong. But were they really? Of course, it’s expected that a percentage of our hypotheses will not be valid, however, it’s also likely that the way we define success or even the segments we select (or don’t) to target could be the drivers of this failure. In this post, I’m sharing a few thoughts on this matter.
Defining success
Since every product is serving its customers in order to achieve a business goal, as product people we - naturally - often obsess with the metrics or actions that are tightly connected with driving revenue growth. As a result, our first instinct while framing a hypothesis is that the cause and effect relationship will instantly influence revenue or an action around it. For instance, we’ll usually say “by allowing my customers do X, they will buy more” or “by removing this obstacle, I’ll increase subscriptions”. However, it’s quite rare that a single action will have such a direct cause and effect relationship with revenue generation events.
In every business, there are input and lagging metrics. And, unfortunately revenue and revenue generating actions usually belong in the latter category. This means that it’s very hard to see a an immediate influence of a single action on them. However, while framing a hypothesis, we very often imply a direct relationship between the cause (by doing X) and effect (I can increase sales). So, a common mistake that we make, is that we define the success of our experiment by tying them into a lagging indicator such as sales, subscriptions or revenue. As a result, we might end up mistakenly considering an experiment failed.
The obvious way to avoid this is by defining the success of an experiment based on an input metric that it’s more likely that we can influence. But this is still not as easy as it sounds. A basic pre-requisite for that is to have a deep understanding of the levers (input metrics) that move the treadmill (lagging metrics such as sales or revenue) and their cause and effect relationship. For instance, for a streaming service, the hours streamed (input metrics) could influence retention (lagging metric).
As much as a no-brainer as that sounds, I’ve come to realise that a vast amount of product teams don’t have a great understanding of the levers that move their lagging metrics. And don’t get me wrong, it’s not an easy piece of knowledge to acquire. It takes months or years of experience on a domain, lots of contact with the users to understand their needs how they operate and even more experimentation in order to understand cause and effect relationships between actions and results.
Of course, having the way in which we measure success is not necessarily the only factor that could mislead us in defining an experiment as failed. The other thing I want to talk about is this post is segmentation.
Selecting the right segment(s)
In cases of products with large (in the scale of millions) and diverse user bases, you may often observe a perceived inability to substantially influence metrics. A common pitfall in this case is the selection of a very broad user segment for the tested hypothesis.
There are many cases where we might get carried away with forming the right hypothesis, or by thinking of the right solution to validate it and we ignore one of the most important things: Thinking of the right user segments that this hypothesis concerns. Next thing we know, is that we launch an experiment in a huge and diverse pool of users and a couple of weeks later we see inconclusive or misleading results.
What we might realise though if we look closer is that this is not true. If we look into more granular segments of our user base, we might realize that our hypothesis was right and we managed to achieve the result that we wanted. We just managed to be successful on a smaller fraction of our users. In general, the larger and more diverse the denominator - the number and most importantly the segments of users - that we use to evaluate the success of our experiment on, the more information can be lost.
So, next time we frame a hypothesis, one of the things that we should pay more attention is to clearly define the segment(s) that we believe that our hypothesis is addressed to. Ideally, we should also make those segments as specific as possible, while of course keeping in mind that they should still have a substantial size in terms of volume that would still allow us to run an experiment and get statistically significant results.
Hopefully, the thoughts expressed above can not only help you get a better success rate for your experiments, but also gain more insights about your users. At the end of the day, we don’t just run experiments to drive better business results. Knowledge acquisition for our users is also vital reason for running them.
Are there any other things that you’ve observed as common pitfalls while running product experiments? I’d love to learn about your own experience.
Past Newsletters
ISSUE #22 - FEBRUARY 20, 2024
Will product managers ever stop spending their time managing delivery?
The time and energy spent on managing delivery is one of the most common sources of frustration for product managers. In this issue, I'm sharing a few..
ISSUE #21 - OCTOBER 25, 2023
From Individual Contributor to Senior Product Leader: The Evolution of the PM Role
The evolving role and responsibilities of product managers as they progress in their careers highlight differences in the areas of progress measurement..
ISSUE #20 - AUGUST 22, 2023
Takeaways after training 150+ AI models in 8 days
The story and the key learnings (not limited to product only) after training 155 AI models during my summer holidays.
The Product Notebook by Manos Kyr.
Once every month, I’m sharing my thoughts on product, growth & entrepreneurship.
Copyright © Manos Kyriakakis