January 20th 2017

5 reasons why your A/B tests are failing

Written by Andy McKenna


Have you been struggling to get successful results from your A/B tests? Do the experiments you run always end up being inconclusive?

Below are five reasons why this might be the case, and some pointers to what you can do to prevent it from happening in the future.

Turn your A/B tests from damp squibs into stellar performers by following these proven techniques.

1. Lack of a hypothesis

As I???ve mentioned previously in a separate blog post, at Slipstream we won???t allow an A/B test to be pushed live unless it has a solid hypothesis. There???s a good reason for this. The single biggest cause ??? by far ??? of failed experiments is the lack of a hypothesis.

A hypothesis provides a focus to everyone involved on what the expected result is. It also holds the A/B test to account, and forces you to do proper research, giving the test the best chance of providing meaningful and valuable results.

Your hypothesis should always be based on a strategic business objective.

2. It was chosen on a hunch

This follows on from the above point ??? your decision of which A/B test to run should be based not on a general gut feeling, but on rigorous data analysis.

To get meaningful results from your optimisation efforts you need to be able to be confident that the tests you run are ones most likely to make the biggest impact. If you are just choosing test candidates on gut feel alone, then you might strike lucky now and again. However it's highly unlikely you're going to have ongoing success.

By basing your test decisions on qualitative and quantitative insights, you're far more likely to test the right things, and prioritise them correctly.

It???s also far more likely that you're going to have a strategic approach that follows a proper test roadmap.

3. The differences between the variations aren't big enough

Don't be afraid to be bold. In CRO there is always the risk of making what???s known as ???meek tweaks???. These are small changes to your site which invariably end up with inconclusive results.

A commonly run experiment is split testing the colour of call to action buttons (e.g. red vs. green). Try to avoid these types of A/B test as much as you can, as they are generally unlikely to make much of a difference.

Put yourself in the customer's shoes, and think about whether the variations you're looking to test would make any difference to you. Would you even notice the difference?

Don't waste time and traffic on tests that will almost certainly prove inconclusive.

4. It didn't run for long enough/was stopped too early

Most testing tools these days have built-in confidence score calculators. Some of the better ones also have test duration calculators. Use them!

Don't be tempted to run tests for just a week. As a rule of thumb, run them for an entire sales cycle. 3 weeks is generally about right, although it will depend on the amount of traffic you get. Also, you don't want to impinge on the volume of other tests you can run.??

The best advice would be to use a test duration calculator (just Google it!) and operate on a case by case basis.

5. You're not analysing the results in enough depth

The majority of testing tools segment test results by a number of different dimensions, including geo-location, new & returning visitors, and device type.

If you have integrated your tool with an analytics program like Google Analytics, you can usually go deeper still with custom segments.??

Digging into segments and audiences gives you far deeper insight into your results. It can turn what appear on the surface to be losing test variants overall into winners within specific segments.??

We've seen examples where a variation gives minimal/no uplift with a low statistical significance on desktop, but on mobile devices sees a significant CVR uplift with 99%+ statistical significance.

Ignoring these segments denies you access to a wealth of invaluable information. This data can act as the cornerstone of your personalisation strategy too.