<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=192061487988692&amp;ev=PageView&amp;noscript=1">

Mastering A/B Testing: Why Should I Run an A/B Test?

Nov 28, 2017 3:28:45 PM  |  BY: Ryan Emond

Welcome to part three of our three part series on the ins and outs of A/B testing. So far you’ve learned the basics of how to set up your test and what you should be testing. Each test will be different and the individual goals you might have will vary from case to case.

If you haven't had the chance to read Parts 1 and 2, I've provided links below:

Part 1: Mastering A/B Testing: When to Run an A/B Test

Part 2: Mastering A/B Testing: What Should I Test?

Below, I've provided the two most common reasons of why marketers run tests along with expert insight on where tests go right and where they go wrong.  

Why Should I Run an A/B Test?

In order to run a successful A/B test, you must have the proper reasoning and intent behing the test. A broad reason for running a test is to optimize for conversions but we must go a bit deeper with our reasoning. Let's dive in with the most common reasons for running an A/B test. 

1. My promotion is underperforming

This is going to be the best reason you’re going to have to start the process towards testing. You are attaching a real concern to something you can actually measure the difference in. Furthermore, by running a test, you can get results that will allow you to make the right decision for your business. There are a couple of potential pitfalls you can fall into though:

Pitfall #1: You haven’t checked your individual promotion performance against industry benchmarks.

Of course you want your promotions to outperform ‘average’, but if your promotion is already outperforming industry averages by 20%+, it may be best to test something else in order to reach your goals. The good news is that your promotion is standing heads and shoulders above your competition. The bad news is that you’re going to have to go in different directions in order to improve further.


Pitfall #2: You aren’t clear about which KPI you want to improve.

If you ‘just want this promotion to do better’ then the chances are you haven’t tied that promotion to a KPI that will impact a bottom line business goal. When you want to improve the performance of a promotion you need to keep an eye on what KPIs will be affected so you can accurately assess a change. If you run a test and you see increased engagements, but the attached KPI was to increase AOV and AOV actually decreased then you need to go back and address that decrease.


Most common KPIs that are attached to promotions are:

1. Overall conversion rate

2. Cart Abandonment Rate

3. Email capture rate

4. Average order value

You should determine which KPI(s) you are targeting with your promotion well in advance of actually launching said promotion to ensure that you can properly track changes.


2. My brand guidelines changed and everything needs to be refreshed

This happens more often than you might think. Maybe you/your boss has decided to give your brand a refresh or maybe you’re launching an entirely new product line. Either way, you need some totally new designs. If you find yourself in this position then your next step is to come up with 2-3 new designs to test against one another. You will not, however, be testing the new designs against the originals in this case.

What you’re looking to do here is take copy or overall UX/UI lessons you learned and try to apply them to your new branding. Basically, you’re starting back over from square one and your tests should reflect you trying to find the best design that works within your new brand without regard for the previous designs.

The reason for this is simple: new branding is expensive and time consuming. Once the decision has been made to go forward with a re-brand then you can’t go back without effectively wasting all of the time and money that went into it.


What A/B Test Success Looks Like

If you've read all three articles in this series, congratulations! You just received a crash course in A/B Testing. Before you take off and setup your first A/B test in your Justuno account, take a look at a recent case study performed by Shopify Plus jewelry retailer, The GLD Shop.

In this A/B Test, The GLD Shop was able to increase conversions by 400%!



Topics: Conversion Optimization, Promotion Strategy

Popular Posts

Subscribe To Justuno

Try Justuno For Free