Suggested readings: Bayesian vs Frequentist Inferences in A/B Testing, Evaluation of A/B Test Results
A/B testing is an experiment that compares one or more alternative versions (variants) of a page against the original, gauging their effectiveness at achieving specific goals (e.g., visitor conversions). It helps marketers understand the performance of different versions of a marketing piece.
The measured conversions range from simple actions (e.g., a banner click, a form submission) to complex actions (e.g., adding an item to the shopping cart, completing a purchase).
This article explains the following concepts:
- How does A/B testing work?
- How do you manage the allocation?
- How do you ensure randomness?
- How do you ensure consistency?
- Best practices
How does A/B testing work?
An A/B test must start with formulating a hypothesis that defines the elements needing improvement and how the changes will positively affect your desired outcome.
During a test, all page variants receive roughly the same traffic and compete for the highest conversion rate or weighted value.
When a user navigates to a page with a running A/B test on a live website, the system assigns a randomly selected variant from the given A/B test variant pool to this user. Each page variant should receive roughly the same amount of visitor traffic during the test period.
How do you manage the allocation?
You can assign users to variants in two ways: manually or automatically. At the same time, web templates use manual allocation, Architect and Email use automatic allocation.
With manual allocation, the classic A/B testing approach, the traffic is evenly split between the variants until a single winning variant is declared with high confidence. Each user is randomly assigned to the variants according to the traffic allocation.
With automatic allocation (a.k.a. dynamic allocation), the highest-performing variant is dynamically observed over time and gradually served a more significant percentage of visitors as more data is collected.
Given the effectiveness, Insider has implemented the manual allocation method for traffic distribution.
How do you ensure randomness?
As the essence of any A/B test, random sampling means that any visitor to your website has the same probability of being chosen to see a variant of your A/B test.
Insider ensures traffic is automatically split across your variants, so each variant receives a random sample of visitors.
Insider utilizes the reservoir sampling algorithm, specifically weighted random sampling, to ensure the highest possible effectiveness and performance.
The weighted random sampling algorithm employs a simple random sample, without replacement, of "k" items from a population of unknown size "n" in a single pass over the items. The population size "n" is not known to the algorithm and is typically too large for all "n" items to fit in main memory. The population is revealed to the algorithm over time, and the algorithm cannot look back at the previous items.
How do you ensure consistency?
Insider stores a special local storage containing the identifier of the selected variant in the visitor's browser. This local storage allows the system to maintain a consistent appearance of the tested page for returning visitors. It also ensures that all conversions performed by A/B-tested users are logged for the assigned variant. Local storage also prevents the campaign from being shown repeatedly.
Best practices
To get the most out of your A/B testing, we highly suggest you:
- Based on your hypothesis, choose the most suitable wording for your campaign so that you can appeal to your users appropriately.
- Choose the correct Call-to-Action (CTA) button in your campaign based on your hypothesis so that you can drive more conversions interactively.
- Ensure the links lead to the right page based on your hypothesis so you can direct your users as desired.
- Based on your hypothesis, choose the most appropriate visuals for your campaign to better grab your users' attention.
- Append your inline personalization correctly so it does not break any functionality on your website and looks harmonious with the other elements.
- Check the landscape of your campaign on Mobile Experience to ensure it fits perfectly.