“Vanity metrics are the numbers you want to publish on
‘TechCrunch’ to make your competitors feel bad.” – Eric
Ries, Author – The Lean Startup
Team Hawk were coming up to the end of their release window
at the end of the quarter. “125 story points,” Pat, their Scrum
Master, proudly exclaimed. “We’re 20 points over last quarter,”
Jessica, their Product Owner, replied. The team celebrated their
increased velocity by planning a trip to the pub that evening.
The efficiency gains were made through ruthless investment in
automation over the past three months. However, not all was as
it seemed. The team’s quarterly objective to increase bookings by
10% through the store’s shopping cart was not met. The features
they released impacted this by just 2% over the quarter. Jessica’s
excitement turned to gloom when the team reviewed the latest
bookings numbers.
Two things happened here. The team was overly focused on an
output metric (velocity). Eric Ries calls this a vanity metric; something which makes us feel good, but has little material impact on
actual outcomes. They had also defined an outcome metric (increase
Bookings), but they had no way of tracking progress on the outcome
metric over the quarter.
Pattern: Leading Indicators
The challenge is that a metric such as Bookings is generally a Lagging
Indicator – we find out after the fact about the impact that has been
made on this, sometimes a long time after the fact when it is too late
to adjust. This lagging indicator is known as a Business Outcome.
What we need in this case are leading indicators tying into this
Business Outcome. These Leading Indicators are described by Teresa
Torres (2021) as Product Outcomes.
Another key point about leading indicators is that they should be
timeboxed to one to two weeks maximum. This would have helped
Team Hawk from our example understand on a more continuous
basis the impact of their incremental deliveries of work. Moreover, it
synchronises nicely with their review cycles – for example, to review
the latest leading indicators at their Sprint Review.
So, what would that look like in practice?
As we can see in the example (above) we still have our Increase
Booking metric as a lagging indicator, but we have plugged a
leading indicator of decreasing drop-off rate between the cart and
the billing page. We know this will feed into our higher level goal
from analysing past data of our checkout funnel.
Pattern: Actionable Metrics
There are two types of metrics related to taking actions: the ones we
can learn from, and the ones we cannot. From an effective metric we
can take insights, discover behaviours, and take actions to improve
our product. Therefore, these are actionable metrics. By contrast, we
cannot learn or take actions from vanity metrics. Whilst our ego may
improve, our product will not.
A mobile app can have millions of downloads but only a few
thousands of active users, or a freemium product can see exploding
traffic growth but barely few conversions to paying users.
Vanity Metrics Actionable Metrics
Hits on webpage Bounce rates
Number of downloads Paying users
Registered users Conversion rate
Social media followers Social media recommendation
Page views App Store ratings
Net Promoter Score Net Fitness Score
Marketing spend Average order value
Raw growth numbers A/B testing
All Actionable Metrics should accomplish the three A’s:
Chapter 2: Measuring it 23
• Actionable: it must demonstrate a clear cause and effect, so
everyone can learn from their actions, realizing how to replicate
the result in the report. The scientific method gives confidence
that the observed behaviour was, in fact, caused by the change
being tested.
• Accessible: it must be simple to read and interpret, and it must
be easy to access the latest data (in a matter of minutes at most).
• Auditable: it must be accurate, enabling skeptics to audit a
report translating the summary numbers in the report back to
the actual users who generated them.
The scientific method to achieve cause and effect for Actionable Metrics. Source
Wikimedia Commons.
The Third Stream Book 