A/B testing is an experiment that compares one or more alternative versions (variants) of a page against the original, gauging their efficiency at accomplishing selected goals (visitor conversions). It helps marketers understand the performance of different versions of a piece of marketing.
The measured conversions are based on visitor actions ranging from simple conversions (e.g., a banner click, a form submission, etc.) to complex conversions (e.g., placing an item into the shopping cart, performing a full purchase, etc.).
This article explains the following:
- How does A/B testing work?
- How do you manage the allocation?
- How do you ensure randomness?
- How do you ensure consistency?
- Best practices
How does A/B testing work?
An A/B test must start with formulating a hypothesis that defines the elements needing improvement and how the changes will positively affect your desired outcome.
During a test, all page variants receive roughly the same visitor traffic and compete with each other for the highest conversion number or weighted value.
When a user navigates to a page with a running A/B test on a live website, the system assigns a randomly selected variant from the given A/B test variant pool to this user. Each page variant should receive roughly the same visitor traffic during the test duration.
How do you manage the allocation?
You can assign users to the variants in two ways: Manual allocation and automatic allocation. At the same time, web templates use manual allocation, Architect and Email use automatic allocation.
With manual allocation — the classic A/B testing approach—the traffic is evenly split between the variants until a single winner variant is declared with a high confidence. Each user is randomly assigned to the variants according to the traffic allocation.
With automatic allocation (a.k.a. dynamic allocation), the highest-performing variant is dynamically observed over time and gradually served a more significant percentage of visitors as more data is collected.
Given the effectiveness, Insider has implemented the manual allocation method for traffic distribution.
How do you ensure randomness?
As the essence of any A/B test, random sampling means that any visitor to your website has the same probability of being chosen to see a variant of your A/B test.
Insider ensures automatically split traffic to your variants so that each variant gets a random sampling of visitors.
Insider utilizes the reservoir sampling algorithm, specifically weighted random sampling, to ensure the highest possible effectiveness and performance.
The weighted random sampling algorithm employs a simple random sample, without replacement, of "k" items from a population of unknown size "n" in a single pass over the items. The size of the population "n" is not known to the algorithm and is typically too large for all "n" items to fit into the main memory. The population is revealed to the algorithm over time, and the algorithm cannot look back at the previous items.
How do you ensure consistency?
Insider stores a special local storage containing the identifier of the selected variant in the visitor's browser. This local storage allows the system to maintain a consistent appearance of the tested page for returning visitors. It also ensures that all conversions performed by A/B-tested users are logged for the assigned variant. Local storage also determines not to show the campaign repeatedly.
Best practices
To get the most out of your A/B testing, we highly suggest you:
- Based on your hypothesis, choose the most suitable wording for your campaign so that you can appeal to your users appropriately.
- Choose the correct Call-to-Action (CTA) button in your campaign based on your hypothesis so that you can drive more conversions interactively.
- Ensure the links lead to the right page based on your hypothesis so you can direct your users as desired.
- Based on your hypothesis, choose the most appropriate visuals for your campaign to grab your users' attention more effectively.
- Append your inline personalization correctly so that it will not break any functionality on your website and look harmonious with the other elements of your website.
- Check the landscape of your campaign on Mobile Experience so that it will fit perfectly.