Onsite Analytics: Metric Definitions

Incremental Revenue

Represents the total extra revenue generated by any Insider personalization in A/B testing. Incremental revenue is one of the important indicators that can help decide if a personalization is a winner. The calculation of this metric is based on comparing the performance of the control group and that of the variant (Insider group).

Revenue (unique) x ΣIncremental (rev) / ΣVar (rev)

Incremental Conversions

The extra conversions gained from the variants when compared to those from the control groups

Total Conv (unique) x (ΣIncr. conversions / ΣVar (purchases))

Conversion Rate (CR)

Indicates the percentage of your visitors who complete the goal for your personalizations. Calculated separately for control and variant groups

Average of Var (CR)= ΣVar (CR) / # of variants
Average of Control groups (CR) = ΣCG (CR) / # of CG’s

Conversion Rate Uplift

Calculated based on the ratio of incremental conversions to the difference between the total conversions and incremental conversions

Incremental Conv / (Total conversions (unique) - Incremental Conv)

Average Order Value (AOV)

Calculated from unique transaction data. Meaning, if the user saw 3 campaigns and made a purchase, then there’ll be 1 unique purchase

Unique revenue / Unique conversions

Revenue from Click

Unique revenue that is generated through clicks

Unique revenue from clicks

Purchases from Clicks

The sum of purchases from clicks for all the variant that has this goal

Σ Variant (Purchases from clicks) -  unique

Clicks

The sum of variant clicks

Σ Variant (Clicks) - not unique

Impressions

The total number of impressions is calculated by summing up the impressions of each variation.
Each time a user encounters the campaign on a different device or browser, we increment the impression count by "+1" even if it's within the same session.
Let's say a user views a campaign on Chrome desktop, Chrome mobile, Firefox mobile, and Firefox desktop, contributing to a total of 4 impressions.
Importantly, this count remains the same even if the same user engages with the campaign across different devices or browsers during a single session.
Refreshing the page within the same session does not affect the impression count, provided the user has already seen the campaign and an impression has been counted as 1. For example, with a re-eligibility duration of 7 days, if a user views the campaign today and then sees it again tomorrow, the impressions remain 1. However, if the user encounters the campaign once more after a lapse of 10 days, the impression count increases to 2.

Σ Variant (Impressions) - not unique

Current Status

Shows the activation status of the variant (active, test, passive). Accordingly, you can sort the results.

-

Start Date

The day your personalization is released or set to Active in DD/MM/YYYY format

-

Incremental Revenue

The extra revenue generated by the variant, calculated by finding the difference between the variant and the control group

Var(Rev) - ((Var (Imp) x CG (CR))x CG(AOV))

Conversion Rate Uplift

Calculated based on the conversion rate ratio of the variant to that of the control group

(Var (CR) - CG (CR)) / CG (CR)

Average Order Value Uplift

Calculated based on the ratio of the average order value of the variant to that of the control group. The difference between the average order value performance of the variant and control group will give this uplift.

(Var (AOV) - CG (AOV)) / CG (AOV)

Impressions (Imp)

The number of visitors who have been in any personalization group. The impressions for the variant and control group are represented separately.

Number of Var impressions
Number of CG impressions

Significance

The global statistical term for A/B testing reports. Determines whether a report is reliable enough to take an action based on it. Significance depends on the size of the experiment, meaning the number of impressions, and the difference in performance, in other words, uplift.
If the significance is near 0%, it means the conversion of the control group is nearly that of the variant group.
If the significance is near 100% and the uplift is negative, the conversion of the control group is more than that of the variant group. If the significance is near 100% and the uplift is positive, the conversion of the variant group is more than that of the control group.

Z-Value=ABS((cr_a-cr_base) / (SQRT(((sales_a+sales_base) / (imp_a+imp_base)) (1-((sales_a+sales_base) / (imp_a+imp_base)))(1/imp_base+1/imp_a))))

Probability to Win

Illustrates the performance of the tested variant in achieving the selected conversion goal compared to other variants, including the control group. It serves as an indicator of the likelihood of the variant's success. The percentage is calculated using Bayesian Significance, leveraging the collected data thus far.

P(A|B) = P(B/A)P(A) / P(B)

Control Group Impressions

It represents the number of CG impressions.
Each time a user views the campaign on a device or browser, we increase the impression count by "+1". This count is not affected by changes in the user's device or browser within the same session. For example, if a user sees a campaign on Chrome desktop, Chrome mobile, Firefox mobile, and Firefox desktop, it would result in 4 impressions even though it's the same user.
Refreshing the page within the same session does not alter the impression count. If a user sees the campaign today and views it again tomorrow, the impressions remain 1. However, after the campaign's re-eligibility duration of 7 days, if the user encounters the campaign again, the impressions will increase to 2. For example, if the user sees the campaign 10 days later, the total impressions will be 2.

-

Variation Group Impressions

It represents the number of Var impressions.
Each time a user views the campaign on a device or browser, we increase the impression count by "+1". This count is not affected by changes in the user's device or browser within the same session. For example, if a user sees a campaign on Chrome desktop, Chrome mobile, Firefox mobile, and Firefox desktop, it would result in 4 impressions even though it's the same user.
Refreshing the page within the same session does not alter the impression count. If a user sees the campaign today and views it again tomorrow, the impressions remain 1. However, after the campaign's re-eligibility duration of 7 days, if the user encounters the campaign again, the impressions will increase to 2. For example, if the user sees the campaign 10 days later, the total impressions will be 2.

-

Incremental Conversions

The number of extra conversions generated by any personalization group is calculated by comparing the variant performance to the control group performance.

Var (Conv) - (Var (Imp) x CG (CR))

Revenue (rev)

Shows the revenue gained from any group of your personalization. Revenue for the variant and control groups is represented separately.

Revenue gained at each group

Conversion Rate (CR)

Indicates the percentage of your visitors who complete the goal for any personalization group. It is calculated based on the ratio of conversions to impressions.

Control group: CG (Conv) / CG (Imp)
Variant: Var (Conv) / Var (Imp)

Average Order Value (AOV)

Calculated based on the ratio of the total revenue gained by any group of your personalization to the conversions of that group

Control group: Var (Rev) / Var (Conv)
Variant: Var (Rev) / Var (Conv)

Control Group Conversions

Shows the number of purchases gained by any group of your personalization. You can check the questions below to comprehend it better:
Q: If I purchase 2 items, does it count as 1 or 2 conversions?
A: It counts as 1 conversion regardless of the number of purchased items.
Q: What if I make 2 separate purchases within the set period?
A: If you make 2 separate purchases within the specified time (conversion period), it counts as 2 conversions. For instance, with an "add to cart" goal and adding 3 different items to the basket after logging in, it will be 1 impression but 3 conversions.
Q: If I make 1 purchase and refresh the success page, does it count as 2 conversions?
A: No, it counts as 1 conversion, even if the page is refreshed.
What scenarios result in 1 impression and multiple conversions?
Similar to the second question, you'll observe multiple conversions under a single impression. For instance, with an "add to cart" goal, adding 3 different items to the basket in one session would be 1 impression and 3 conversions.

Number of purchases gained through the control group.

Variation Group Conversions

Shows the number of purchases gained through the variation. You can check the questions below to comprehend it better:
Q: If I purchase 2 items, does it count as 1 or 2 conversions?
A: It counts as 1 conversion regardless of the number of purchased items.
Q: What if I make 2 separate purchases within the set period?
A: If you make 2 separate purchases within the specified time (conversion period), it counts as 2 conversions. For instance, with an "add to cart" goal and adding 3 different items to the basket after logging in, it will be 1 impression but 3 conversions.
Q: If I make 1 purchase and refresh the success page, does it count as 2 conversions?
A: No, it counts as 1 conversion, even if the page is refreshed.
What scenarios result in 1 impression and multiple conversions?
Similar to the second question, you'll observe the case of multiple conversions under a single impression. For instance, with an "add to cart" goal, adding 3 different items to the basket in one session would be 1 impression and 3 conversions.

The number of purchases gained through the variation group.

Effective Days

Shows the number of days the variant had more than 10 impressions.

-

Clicks

Any click on a campaign except for white spaces and the close button is counted as a click.
If you see a campaign and click it, it's counted as a click. If you visit another page, see the campaign again, and click it, no click is logged until you fall into the re-eligibility duration of the campaign again. 

-

Unique Revenue

This value is derived from unique transaction data. In other words, a user viewing three campaigns and making a purchase results in a single unique purchase.

-

Unique Conversion

This value is calculated from unique transaction data. In essence, if a user views 3 campaigns and makes a purchase, it will be counted as 1 unique purchase.

-