“Measurement is fabulous. Unless you’re busy measuring what’s easy to measure as opposed to what’s important.” – Seth Godin
Daniel, the VP of Product, arrived in Team Purple’s Sprint Review. There was a sense of anticipation in the room as the team knew he was an influential stakeholder on the initiative they had been working on around the shopping cart redesign. As usual, the team demoed working software they created over the past sprint. Jack, the team’s Product Manager shared the latest customer feedback from focus groups on the feature. At the end of the Review, Mary, the team’s Scrum Master proudly shared their metrics dashboard.
“As you can see here, our release burnup chart shows we’re on track for sprint 15 to release our next increment, our velocity predictability is at 96%, we have 2 minor bugs in the backlog, our automation coverage is at 92%, and our team happiness metric is trending up, ” she said proudly.
“That’s great, but how is our conversion rate trending since we launched the last update?” asked Daniel.
The team stared blankly at each other. All of the metrics they focused on were output metrics, which were also important. However, aside from customer feedback from Jack, they never reviewed outcome metrics. They had discussed it in a previous retrospective, but came to the conclusion that it would take time away from feature work and it was also too hard to measure changes in customer behaviour.”
The team here had fallen into the trap of being overly output metric-focused. These metrics are generally easier to gather and help understand delivery agility, not necessarily customer or business impact. Delivering at speed with quality and predictability are all great, but to what end? We must focus on effectiveness metrics first, followed by efficiency metrics.
The ultimate measure of success these days is no longer working software, but instead software which delivers on customer and business outcomes. Customer feedback is great, but qualitative feedback must be balanced with quantitative product data.
Pattern: The dashboard hack
“If you measure it, it will improve” – Seth Godin
Focusing on the right outcome metrics to gather is difficult. Even more difficult is building into your culture to become data-informed. We believe that a good first step is to pick just one outcome metric and make it visible.
Begin by understanding what is causing your customers the most pain and consider how you would measure their pain is being relieved. Begin measuring this. Then, add this one metric to your product team’s dashboard alongside all of the existing output metrics. Treat this metric as an equal class citizen, no matter how painful it might be to see how it is trending. Reviewing this as part of your existing team meetings will serve as a forcing function for the team to review it on a regular basis and discuss how to improve it.
The Third Stream Book 