Do you remember when you were in school and you had to do a science fair project? Well, a lot of the principles that we learned still apply to create a proper A/B test. Ultimately, running A/B tests can increase the performance of your refer-a-friend campaigns and provide valuable learnings for future marketing initiatives.
Following the Scientific Method
To run tests that yield information you can trust, try using the scientific method. It follows six steps:
- Make observations
- Formulate a hypothesis
- Conduct an experiment to test the hypothesis
- Evaluate the results
- Accept or reject the hypothesis
- Determine whether or not to test a new hypothesis
What is a Hypothesis?
Your hypothesis is your theory. For example, a hypothesis to an A/B test could be that you think green call-to-action buttons have better click-through rates (CTRs).
Another essential part of running an effective experiment is how variables play a role. A variable is anything that can be changed or controlled in your tests. If we wanted to test the call-to-action hypothesis, a variable in this experiment would be the color.
Types of Variables
There are three types of variables in the scientific method: controlled, independent, and dependent variables.
- Controlled Variables, also known as constant variables, do not change. Building on the CTA example for this experiment; the copy and the button size and shape should all match in both tests.
- Independent Variables are the factors you are changing. By having one independent variable, it makes it simpler to measure and analyze your test data. In our example, the color would be the independent variable because it’s the only element that changes in the experiment.
- Dependent Variables are the variables you are watching. In this case, the dependent variable is the click-through rate because you are looking to see if the green CTA has a higher click-through rate than a different colored CTA.
Various Types of Experiments
There are different types of test that you can run. However, A/B tests are useful because you are testing one variable and it’s definitive how that factor played a role in the performance of your marketing campaign or advertisement.
A/B tests measure one constant variable against different possible choices for that variable. For example, you could apply this to a landing page where you could compare hero images and see if one image performs better than another.
Multivariate tests are experiments that test multiple elements at a time. By running these types of tests, the goal is to learn which factors together make the most significant impact. On a landing page, you would test both images and copy. However, unlike an A/B test, you won’t be able to definitively say that image A generated more views than image B because you were testing multiple variables at a time.
A structured plan for your A/B tests can have a very positive impact on your business, Econsultancy ran a survey and found that 74% of respondents who had a plan improved their sales.
Create Your A/B Testing Plan
It’s crucial to build a testing plan that is structured and yields fruitful informative results. Here are seven steps for you to follow to create a strong A/B testing plan (our testing plan follows the scientific method):
1. Make observations
Look at your campaigns and see if there are patterns of notably high or low performance. Think about the tests that would help prove why the performance rates or trends were occurring.
2. Come up with a list of tests
Now that you’ve observed your marketing activities, come up with a list of tests that you would like to complete. To determine which tests you should run first, consider the ones that will make the most impact on your business, how important is one test over another, and the ability to execute the test.
3. Select your test and create a hypothesis
Your hypothesis should state your theory on the outcome of the test. A prediction will help ensure your research has a clear purpose and either prove or disprove your theory.
4. Run your A/B test
Now that you have a theory to test, you need to run your experiment. A/B test are helpful because they can handle your traffic distribution as well as determine which variable had favorable results.
For example, the Talkable platform allows users to A/B test and determine the traffic distribution. We will also tell you the winner of the test based on when the experiment reaches statistical significance. Talkable determines when a test has reached statistical significance by accumulating large enough sample size, which usually takes around two weeks. Also, the platform’s dashboards show the percentage of confidence that the winning version generated too.
When you create your A/B test, remember to test one variable at a time. For example, if you’re testing two emails to see which one drives the most traffic to your landing page, there shouldn’t be another A/B test running on the page. You’ll never determine whether the increased performance was due to the emails or the landing page.
5. Evaluate the results
After you’ve run your tests, now comes evaluating the results. You can calculate results based on the statistical significance or pick a sample size that makes sense for your business or you need to achieve. Talkable’s software picks a winner based on the percentage level of confidence. By analyzing your results, you’ll be able to accept or deny your original hypothesis.
6. Accept or deny your hypothesis
Now is the time to determine whether your initial theory was right or wrong. Once you pick to accept or deny your hypothesis, try and apply your learning to other parts of your marketing activities, channels, and throughout your website.
7. Run more tests
At the end of your test, you should conclude your findings or decide that you still need to gather more information by doing another follow-up test. Regardless of your choice, you always will be running more tests since the web is constantly influx, changing and improving all the time. What worked in the past won’t necessarily work in the future!
Running an A/B on Talkable
With a testing plan in place, it’s easy to execute tests with Talkable. Here’s an example of someone testing the copy on their offer. In the Easy Editor, click the create A/B test button to the variable you are testing.
Since we are testing copy, we’re creating a test under the copy tab. However to use other variables like images, color, and configurations select the corresponding variable at the top of the panel.
Next, you’ll be taken to our create an A/B test page. Here you can name your test, add in the copy you would like to test, determine the traffic split, apply to the test to the campaign(s), and preview it.
Now that your test is live, you can view the A/B test report. Talkable will start to collect reporting information. While you’re running tests, you can keep track of them in your dashboard tab. You can easily access A/B test reports here as well. The A/B reports offer information about the status, progress, type, and impressions generated from your experiment.
Also, the report offers you different views based on the hypothesis, the funnel, and sales. Here you can also see the results of your test. Below you can see each variable and how many offers, sign-up and even sales the test help to generate.
Talkable will tell you here which variable the platform thinks is the winner based on the percentage of confidence. If you agree, then you click the set as a winner button. Now the test is done, and you’re ready to run more tests.
Running A/B tests is critical to launching any healthy marketing campaign. By following a structured testing plan, marketers can learn what works best for their audience and continue to optimize their campaigns. Using A/B testing tools can make habitually running tests easier because they provide in-depth analysis and select variant winners. Ultimately, optimizing marketing activities will result in better performance and ultimately better results for your business. Take a look at the results our team found from our three client wide A/B tests.