“There is surely nothing quite so useless as doing with great efficiency what should not be done at all.” – Peter Drucker
What is the final purpose of a product? Quite simply, it is to generate value for the organisation which is sponsoring its development. The final value usually is measured in terms of something like revenue. So, measuring medium and long-term metrics such as this is important to understand the product’s impact.
The challenge many product teams face is they may not have any idea if what they are building is really being impactful. They’re focused solely on output delivery, and are not understanding the connection between this output to user behavioral changes and ultimate business impact.
Last but not least, measuring impact is about what really matters in the end: business value. A product exists to achieve a goal. Of course, to improve its customers’ lives in some way. After that, for the organisation, the product is the way to accomplish revenue, user growth, retention, or even recommendation.
When the product’s development is not connected to a higher level impact goal and metrics, we often see a few wastes:
- Inflated product: We don’t know when we achieve the maturity of the product on its lifecycle, so we keep evolving it creating new features, things that end users don’t want, and which can actually cause friction for them.
- Product team disengagement: Lack of purpose happens when someone is not seeing how their work is making difference. Not just feeling, but really seeing how their work contributes to the organisation’s goals. Recognising impact of their work is one attribute for effective teams according to Google’s Project Aristotle research.
- Not exposing product impact to the company: Stakeholders and the entire organisation can lose trust that product outcomes will be achieved, removing budget or people from the product’s development.
- No strategy to produce the proper impact: If we don’t have impact metrics, how can we ensure the product is really impacting the results of the organisation?
- Feature Factory: Our product development cycle can be just a Feature Factory, producing a lot of customer frustration, business impatience, team burnout or apathy. It’s a term coined by John Cutler, and is very related to impact metrics. Check out John’s tweet which first coined the term to get a sense of the pain many of us have experienced:
Impact metrics are what connect product development with the business strategy. However, it can take a long time to achieve impact metrics, and this is often cited as a reason they are not measured. Thus, we know product teams should not set a goal to achieve impact metrics in a direct way, because they are lagging indicators.
Product teams are driven by leading indicators (outcomes), whilst lagging indicators (business impact) are measured to inform the results of all work that has already happened.
Pattern: Impact funnel metrics
What happens after we ship? We need impact metrics in order to observe a shift in actualised value. Even though it can take months to see a result, it helps the close the Three-Stream learning loop.
Startup Metrics for Pirates: AARRR
The simplest impact metric we can use is revenue. However, it can often be too isolated to derive what is driving revenue. Before achieving revenue, we can map previous steps to this final impact: How many users are we acquiring? How many users are really using our product? How many users are coming back again? How many users are referring and promoting our product? How many users are paying for it? These metrics can be represented as a funnel.
These metrics are not just useful for startups, but for any product development effort we want to measure, learn and act upon. It was developed by Dave McClure to help startups to find their product market-fit, grow and scale.
The acronym AARRR (yep, just like a pirate shouting!) means:
- Acquisition: how many users are coming to see the product?
- Activation: how many users are starting to use the product?
- Retention: how many users are returning to use the product?
- Referral: how many users are recommending the product?
- Revenue: how many users are paying to use the product?
We know the final result is revenue, but the product strategy is to convert the cohort of acquisition users to activation, activation to retention, retention to referral, referral to revenue. Thus, as seen above, this can be visualised of as a funnel; a strategy of your product to get more value to the market and the business.
The Third Stream Book 