”In God we trust. All others must bring data.” – W. Edwards Deming
Team Creed huddled around their dashboard to review their latest live product data. The atmosphere in the room felt heavy.
They had implemented a new landing page for their U.S. market for the cash cow product of the company. Users had gushed over the new design in prototype testing. They had commented that seeing all the benefits of the product on the landing page would entice them to buy it. They complimented the slick new UI, and friendly new copy. The new experience passed all usability tests with flying colours.
There was one big problem, however – since the new landing page launched, bookings were down. Jim, the team’s Product Manager, struck a disturbed look. The reflection of the negative $ numbers from the screen was in his eyes. He had to make the call – it was time to roll back! The page was pulled and never saw the light of day again.
There was a sense of frustration from the team in the air. Emily, the researcher, couldn’t understand why this page was not succeeding out in the wild. Anton, one the engineers, was annoyed that the team had spent 2 weeks building this thing, and now it was being thrown away. The stakeholders were all disappointed that the opportunity and sure fire bet that Jim had talked to them about had failed in a big way.
Prototypes are initial cheap bets to reduce risk, but they do not mitigate risk completely. The bottom line is this – you have not fully in/validated your hypothesis until it is in real users’ hands and they are giving you money.
This is why we would recommend, in addition to prototyping, that your Product teams consider using techniques such as A/B testing, to validate hypotheses with real users in a live environment.
Pattern: Real-time betting using A/B testing
These days, it is hard to find a product team who don’t at least refer to A/B testing in some form or other. There are a ton of tools on the market to do this, such as Google Optimize and Adobe Target, however we rarely see A/B testing done well. Some people we meet are put off by the thought of having to make sense of data. Flashbacks of doing Maths exams come rushing into Product people’s heads! However, it doesn’t need to be like this.
We also have experienced Optimisation or Analytics teams as separate parts of the organisations, rather than as a skillset embedded within empowered Product teams. We have worked with teams who have learned the basics, and really excelled with that knowledge alone.
The patterns we will share with you here are intended to help you get bootstrapped with A/B testing. They are not intended to be comprehensive, there are other great books out there, such as A/B Testing: The Most Powerful Way to Turn Clicks Into Customers (2015) to help you dive deeper. We want to give you enough concrete examples to get you curious and on your way.
Types of tests
It’s important to recognise firstly that there are different types of tests that fall under the conversation around A/B tests.
Classic A/B test
With this type of test we are pitting a Challenger against a Champion. Its Foreman vs Ali, Drago vs Balboa, Warrior vs Hogan, Pepsi vs Coke – you get the drift.
For a landing page example, we want to test our call to action and product details area with new content for the headline and buy button, vs our current experience. The goal is to see how this will impact how many items are added to our cart, which in turn will impact conversions. So we set 10% of our traffic to the new experience, and 90% to the current, control experience. The users reaching each experience are not aware that they are seeing a different variant. Based on this, we can understand the different behaviours from the two cohorts of users.
A/B/N test
Similar to A/B tests, instead of having a Champion and one Challenger, we have multiple challengers.
So Team Creed would have tested multiple different landing page modules against the control experience, to understand how each experience impacted user behaviour.
A famous example of A/B/N testing was when Marissa Mayer of Google tested 40 shades of blue to determine the right color for the links. Mayer’s team set up an A/B/N test with 40 different shades of blue displayed randomly to each 2.5% of visitors and measured which colour earned more clicks. That folks, was how the blue colour you see in GMail and on the Google results page was chosen!
Now, don’t for a minute suggest you test 40 different variations of a colour in your own experience, but you can see how perhaps 3 or 4 different options could be tested to help validate your hypothesis.
Multivariate test
This type of test is distinct from an A/B/N test in that we are testing multiple variables within the same test. We are trying to figure out which parts of the test are causing behavioral change for our users.
In the case of Team Creed’s landing page, if they wanted to understand which was the optimal mix of 3 different headlines, 3 different sets of benefits and and 3 different button calls to action, they could run a Multivariate test to understand which combination optimally changed user behaviours in such a way that impacted items added to cart. In this example, we would be testing lots of different combinations, and the tool would present us with a configuration which reached a high enough confidence level to validate our hypothesis.
Connecting the First and Third Steams
Whatever A/B testing approach you use, remember that ultimately we are trying to reduce risk on our bet. A prototype is a great way to do this in the First Stream, but it often requires an accompanying test in the Third Stream once your product meets the acid test of direct contact with its market.
The Third Stream Book 