A new report says A/B testing is one of the biggest drivers of incremental growth for publishers. Produced by online media-business magazine Digiday and pub-tech provider Piano, ‘The publisher’s guide to A/B testing’ presents an overview of the key aspects of the technique and offers advice to publishers on how to implement it effectively.
Takeaways
- Looking for ways to optimize and incrementally improve subscription conversions is essential for publishers with reader revenue programs in place. Understanding audience behaviors, how they respond to specific content types and particular subscription offers, helps shape effective subscription strategy.
- A/B, or split testing, measures two or more elements of a web page or email with different audience cohorts to determine which version delivers the best outcome. Publishers are increasingly using the approach to boost engagement and revenue and the new guide highlights how publishers can make the most of it.
- The report advises publishers on setting up and running A/B tests on their own publishing platform and identifying the best KPIs. It also looks at the numbers behind this method of testing and how to find partners and resources to assist.
Mitigating risk
- According to Alexander Kreybig, Director of Strategic Services at Piano, growth is achieved by changes in setup, pricing or user experience, but every change represents a risk. He said:
A/B testing mitigates the risk and enables you to test that change and see if it brings statistically conclusive results.
- With A/B testing, as with all methods of analyzing audience behaviors, there are generally multiple variables relevant to achieving desired outcomes and publishers should be careful to focus on the factors critical to their overarching goals.
- Kreybig describes a price test that involves different price points, but also different colors. He said:
Did you get the results because of the price change or the color change? There are plenty of examples like this where you don’t understand the results at the end. Think simply. Test a single thing — what you truly think will make a difference. Not five, one.
Testing methodologies
The report says there is no single ‘right’ way to conduct A/B tests. But it does recommend that publisher familiarize themselves with the ins and outs of testing methodologies on modern testing platforms, especially the metrics that can reflect growth.
Kreybig recommends working with large partners or technology solutions that have industry benchmark data available. Without that data is is difficult to know whether your performance is good or bad. He said:
When you have a large enough partner who has a few hundred sites, that partner can help you understand your results in a broader context.
Case studies
The publisher’s guide to A/B testing also features case studies to illustrate the approach in action.
- India’s largest tech media platform, Inc42, used insights garnered from A/B testing to increase its average monthly sales volume by 100% and customer retention rate by 43%.
- Spanish digital publication El Confidencial (EC) increased subscriptions by 60% after using A/B testing to judge the effectiveness of placing premium articles on its homepage, more engaging call-to-action buttons, and the revenue benefits of discounted prices.
One other point made is the importance of timing, especially knowing when it is time to stop running tests and use insights to optimize your subscriptions strategy and ‘move on to the next test’.
