“Tell me how you measure me, and I will tell you how I will behave” – Eliyahu M. Goldratt
Metrics are a way to know in what direction something is going, in order to take informed next steps. Quantitative metrics enable clear-cut numerical analysis with direct indicators, while qualitative metrics enable subjective study about some user behaviour, opinion or expectation. These two types of metrics complement each other.
Measurements are everywhere we look in life. How many minutes left on my drive to work? How much did we fundraise during our charity event? What was my step count today? What odds is the team we want to back in the match? Ultimately, they help inform our decisions as to what a good next step is.
From a Product perspective, one analogy we like to use is around placing bets. If we were betting on the English FA Premier League football match, we might walk into a bookmaker and review the odds of the next match. We bet money on the favourite. It is the top of the league team playing at home against the bottom side. 90 minutes later, our money is down the drain, as a outsider pulls off a 2-0 win. An eruption of cheers emanates from the back of the room, as a group wins on their bet on the outsider. Why did the favourite not win at such attractive odds? It had won its last 5 matches after all. Well, according to research, the home favourite only wins 55% of the time. As we can see here, being data-driven doesn’t guarantee the outcome we are seeking (to win) from our bets. Even worse, we make this same mistake with our products: 95% of new products fail according to Harvard Business School professor Clayton Christensen link.
Instead, we need multiple data points – some qualitative and some quantitative – along with plain old gut feel! In the match example here, the group who won also monitored the away team’s recent player transfers and the fact that the home team picked up some injuries, having played only 3 days beforehand. This additional data influenced their decision about where to place their bet. From a Product perspective, this means when we are placing our bets, we need leading and lagging indicators, qualitative data, along with plain old gut feel (we prefer to think of this last one as product sense).
This chapter will call out measurement antipatterns to look out for with a short story each. We will discuss some light theory as to why we believe each is an antipattern, followed by some concrete patterns and practices to experiment with.
Antipattern: Would you recommend it?
Antipattern: I’ve got 99 metrics, but outcomes ain’t 1
Antipattern: Lone-wolf metrics
Antipattern: No value-driven metrics
Summary
In this chapter, we discussed when placing Product bets, why we need different types of measurements to inform our next best decisions.
We explored:
- Why vanity metrics might make us feel good, but won’t help us with outcomes.
- The difference between leading & lagging indicators, and why metrics should be actionable.
- The trap of using NPS, and what to use instead.
- How to hack your team’s shift from outputs to outcomes by beginning to infuse outcome metrics in their information radiators, such as dashboards.
- Why standalone metrics are dangerous, and instead, why we need a constellation of connected metrics. Models such as Driver Trees can help us identify those connected metrics.
- Impact metrics inform the product strategy to engage the product team and the business to produce what matters.
In the next chapter, we will discuss how to take these measurements as an input into our learning cycle.
Next Reading: Chapter 3: Validating Learning
The Third Stream Book 