<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=192061487988692&amp;ev=PageView&amp;noscript=1">

Mastering A/B Testing: When to Run an A/B Test

Oct 12, 2017 12:00:00 AM  |  BY: Ryan Emond

Mastering_AB_Tests.png

By now most marketers have at least heard that they need to integrate testing into their process for developing new promotions and advertising.

You have probably heard the terms ‘A/B testing’, ‘A/B/n testing’, or ‘Multivariate testing’. You may have even done a little bit of research into what each of these things are. Let me save you some time:

  • ‘A/B’ and ‘A/B/n’ testing are essentially the same thing. They are both testing one specific variable change to see the impact of that change.
  • A Multivariate test is any test wherein you are testing the impact of multiple variable changes are once. The hint is in the name itself: multi (many) variate (variables).

We'll cover what types of tests you should be running in a later post, but for now we wanted to make sure you have a clear understanding of the terms involved and a broad understanding of what they mean.

In this series of posts I’m going to go over the details of A/B tests. Specifically the 3 W’s of A/B testing. This post covers everything you need to know about when to run an A/B test. Lunch jump into it!

 

When to Run an A/B Test

The first step in designing any test is to figure out when it is appropriate to test.

It’s easy to say, ‘Always'. You should always test everything’.

I sometimes still fall into this mindset myself, but before each test it’s important to take a step back and ask yourself a few questions:

How many visitors are going to see this promotion over the next week? Two weeks? Month? 

The answer to this question is incredibly important. In an ideal world you will need a few thousand points of data (read: visitors actually running through your test). In more realistic terms if you have at least a thousand visitors that come through your test you can be reasonably confident in the results of your test.

So, your first step is to figure out how many visitors will predictably funnel through your test per day. This number should be easy to get from your Google Analytics or Justuno dashboard if you have your Analytics account linked. 

 Google_analytics_report.png

 

Keep in mind here that we are assuming that you want to test a promotion on your homepage and as such this promotion will be seen by a majority of the visitors to your site. If you are targeting a deeper page on your site you will need to look at your Analytics data for that specific page. This, again, should be easy, but is important to keep in mind.

Once you have your average per day visitor count you can calculate the minimum time you’ll need to run your test.

Make sure that each test you run starts and ends on the same day of the week at the same time of day. This critically important as it allows you to control for fluctuations in traffic and user behavior from day to day. What this means is that, generally, your minimum testing window will be one week.

Also be sure not to start a test that will run through a holiday as the fluctuation in traffic and buying patterns with throw off your data and invalidate the test.

Generally speaking the maximum time you want to run an A/B test will be one month. The reasons for this cap are based around resource allocation. If it takes longer than a month to reach minimum viability, or about 1,000 visitors, in your data then you need to think about running a different type of test.

Once you have established your average per day traffic flow you can then figure out the length of time you need to run an effective test. See our handy chart below for guidelines to get started:

 

 # of Visitors per day

 Minimum length of test

0-100

14 days (2 weeks)

101-250

14 days (2 weeks)

251-500

7 days (1 week)

501-1000+

7 days (1 week)

 

I’m sure you noticed that even at ~500 visitors per day the minimum testing length is still 7 days. Why?

Because even if you have 10,000 visitors per day your site will experience different user behavior on different days of the week. By starting and ending your test on the same day of the week you can help smooth unusual user behaviors and patterns.

 

Summary of ‘When’: Dos and Donts

Do: 

  1. Get your per day traffic
  2. Use traffic numbers to determine length of test
  3. Start and stop your test on the same day of the week

 

Don’t:

  1. Start an A/B test on a weekend
  2. Run your A/B test for longer than a month
  3. Start and stop your tests on different days, at different times

 

Keep Reading!

This is the first part in what will be a 3 part series on our ‘how to’ for A/B testing.

Our hope with this series is that by the time you have finished reading part 3 you will have a taken your first steps into A/B testing that is based on sound practice.

Thank you for reading and keep your eyes peeled for Part 2: What to Test!

Topics: Conversion Optimization, Promotion Strategy

Popular Posts

Subscribe To Justuno

Try Justuno For Free

SIGN UP FOR FREE