<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=192061487988692&amp;ev=PageView&amp;noscript=1">

Mastering A/B Testing: What Should I Test?

Oct 30, 2017 11:22:09 AM  |  BY: Grant Thomas

If you're just starting this article and have not read part 1 of this series, I recommend going back and reading Mastering A/B Testing: When to Run an A/B Test before jumping into this article. 

Now that you have a better understanding of how to time your A/B testing periods, we can dig into the most important and most commonly misunderstood aspect of testing - how to structure a reliable A/B test.

Elements of a Promotion

1. CTA - The Call to Action

2. Tag Line - Copy to grab visitor attention

3. Offer Copy - The copy that describes your offering

4. Offer amount - The tangible value of the offer

5. Image asset - Image used to represent offer, brand, etc. 

6. Number of steps - How many steps it takes a visitor to complete the engagement

Numbers 3 and 4 are separated because you can change the specific offer amount without actually changing the surrounding copy. Here's an example:

‘Start today to get 10% off your subscription!’


‘Start today to get 15% off your subscription!’

In the above the example, the ‘offer copy’ is the text that your ‘offer amount’ is wrapped in. This distinction is important to keep in mind as you move forward into creating actual tests.

Each part of a promotion will generally be weighted differently in terms of significance. Offer Amount is almost always going to have the most significant weight. If you think about this in practical terms you can see why.

For example, would you rather have $5 off a premium car wash or a FREE premium car wash?

All other things being equal, more people are going to be more willing to get the free car wash than just $5 off.

With this example in mind, let’s take a look back at the list of promotion elements. However, this time they are ordered by most important to least important:

1. Offer Amount

2. Number of Steps

3. CTA

4. Image Asset(s)

5. Tag Line

6. Offer Copy

The above list is much more useful and also much more open to debate, but for now use this as your starting point and adjust the rankings as you proceed with your testing. Each site will attach different significance to different elements so I recommend the following easy task that will help make building your testing process much easier.


How to Properly Set Up a Test

The most important first step in setting up any test to optimize for conversions is to identify the KPI you want to affect. This could be reducing cart abandonment, increasing email sign-ups, increasing engagements, or any site-specific KPIs that you and your team have identified. Once you have identified your KPI you can then take the component list from above and start formulating hypotheses about which of these elements will best help you improve the KPI you have targeted. Let's sketch this out for you with an example so you can see how this is done:


Step 1: Identify KPI



In this example there is a lot going on and a lot to unpack, so let’s start by breaking it down using the list from above.

1. Offer Amount - In this particular promotion there are actually two different offers being pushed. 20% off and free shipping.
2. Number of steps - 1
3. CTA - none, simply clicking a code isn’t a clear action
4. Image asset - none
5. Tag line - Instant savings
6. Offer copy - take (offer amount 1) the items in your cart now + (offer amount 2)

Now you get a better understanding of why a priority list is important. You can visually break down each part of a promotion and have a clear understanding of all the different moving parts.

Let’s start with identifying a hypothetical KPI for this promotion - increase average order value. This is a clear bottom-line KPI that is measurable.


Step 2: Identify and Hypothesize

In this step you need to identify the elements that you believe will lead to having the most impact on the KPI you identified in Step 1. Luckily, because part of Step 1 is also identifying and weighting each element of the promotion this part should be easy. As an example, a hypothesis would look like this:

I believe that by changing the Offer Copy to include a minimum order value and increasing the visibility of the Offer Amounts that we can increase AOV.

As you can see a typical hypothesis is not going to be overly complex, but it is going to be very specific about actions and outcomes.

Step 3: Create a variation

Now that we have a hypothesis in hand we can take the identified actions. Luckily with Justuno this part is usually very simple.

The variation:


The actions identified in step 2 have now been iterated on and now I have a viable variant I can set up to run a test with. The Offer Amounts have a consistent bolding as well as the minimum required. A very quick read should, in theory, convey the information of: 20% off, free shipping, I have to buy $100 worth of stuff.

I have also visually de-prioritized ‘instant savings’ because that text is largely irrelevant to my goals. While I could have made many additional changes that would have been largely out of scope for most forms of A/B testing. In most cases, you will want to make sure that you are testing changes in one thing - which leads me to the last step.


Step 4: Test and iterate further

Now that you have your variation, you can set up your test. The length of time you should run a test will entirely depend on how long it will take to reach statistical significance - which is just a fancy way of saying, ‘the data you’re getting is 90% reliable’. There are some formulas you can use to calculate statistical significance, but in most cases you will need at least 1,200 impressions in order to have reasonable confidence in your results.

Remember, The minimum length of time you want to run a test is one full week. Your test needs to start and stop on the same day and, ideally on the same time. If it’s already late in the day, then schedule your test to start early on the following day. Another note, it is best practice to avoid starting a test on a Friday.

With those general rules in hand, you can go into your Google Analytics and look at the page that your promotion will live on and find the information you want. Go to the Audience tab and set the date range to look at the last 3-6 full months then sort by week. Each data point available on the chart will be a 7 window into your traffic.

Enter each of the data points into a spreadsheet (Google Sheets or Excel will work equally well) then average those numbers. You now have a reasonably accurate average weekly traffic flow for your site. If that average is at least 1,200, then you can safely set your test to run for one week. If that average is less than 1,200, then you need to start adding weeks onto your test.

A lot of other CRO papers will say not to run a test for longer than a month, but realistically 3 weeks is more accurate. If it’s going to take you longer than 3 weeks to hit the minimum you need to hit significance, then you will want to go much bigger with your test and test a lot more variable changes.


Run your First A/B Test!

As your first test starts to run, you should go back to Step 2 and start over to get you and your team into a continuous cycle of improvement.

That wraps up Part 2 of our 3 part series on A/B testing. By now you should have a good grasp of when and what to test. In part 3 we will discuss why testing is important.

Interested in optimizing your pop ups and promotions? Give Justuno's A/B Testing engine a go today. All features are available on the Justuno Free plan!



Topics: Conversion Optimization

Popular Posts

Subscribe To Justuno

Try Justuno For Free