Auto Optimization

Auto Optimization aka Multi-Armed Bandit enables automatic reweighting of variation groups based on performance

Auto Optimization is a SiteSpect feature set at the Campaign level that compares the performance of Variation Groups as a Campaign progresses and adjusts their weighting (Frequency) according to a collection of settings. SiteSpect measures how well each Variation Group performs using the Campaign's Key Performance Indicator (KPI) compared to the Control Group. As the Campaign continues, SiteSpect examines the performance of Variation Groups and adjusts the traffic they receive based on your settings. Variation Groups that don't perform well receive less traffic than Variation Groups that perform better than expected.

Note: Optimization is available only in the MVT Campaign Builder.

At a high level, you can choose between the following settings:

Off – SiteSpect makes no recommendations.
Recommendations – SiteSpect makes recommendations about weighting Variation Groups. You can choose to accept or reject the recommendations.
Automated – SiteSpect manages the weighting automatically.

There are a number of thresholds that determine when optimization takes effect. Some of those thresholds are:

number of visits to the Campaign
number of days since the Campaign started running
significance above a certain percentage

All of these settings are configurable on a per Campaign basis.

Note: After the initial threshold SiteSpect makes recommendations once per day.

Turning on Optimization

To establish settings for Optimization:

Edit the Campaign.
On the Name & Status tab of the Campaign, scroll down to the Optimization setting.
Select either Recommendations or Automated, depending on the level of optimization you require.
Above that field, click the Options button under the Critical Metric field. Optimization options are available only after you turn Optimization on. The following dialog opens. The top part of the dialog allows you to determine the Critical Metric, which is likely already selected. The bottom part lists the Optimization settings.
Complete the fields in this dialog using the definitions in the table below.

Setting	Meaning
Minimum number of days since start or purge of Campaign	Automation does not run until this threshold is met, which is the number of days since you started or purged the Campaign.
Minimum sample required for analysis	Automation does not run until this sample size is met, which is a number of visits to base the comparison on.
Minimum significance level (1-100)	SiteSpect does not re-weight Variation Groups unless their results are at least this significant.
Maximum re-weight amount (0-100)	This number plays two roles: It is a multiplier for the amount to re-weight each Variation Group. The lower this number is, the smaller the weight changes are. It is also a hard limit for how large a change can be. If a re-weighting exceeds this number, it is reduced to this maximum instead.
Minimum weight for Controls (0-100)	SiteSpect does not re-weight a Control Variation Group beneath this percentage of traffic.
Critical Metric Goal: Highest value above Control	Lets the user specify whether a Variation Group is doing well when the Critical Metric has a high value.
Critical Metric Goal: Lowest value below Control	Lets the user specify whether a Variation Group is doing well when the Critical Metric has a low value.

Summary of Calculations

SiteSpect optimizes using Variation Groups, but calculates the change per Variation within each group, using the following equation:

change = sign * confidence_factor * delta_factor * max_reweight_percentage

From the formula	Meaning
sign	If the Variation sees higher average values for the Critical Metric than for the Control Variation, this is 1; otherwise, it's -1.
confidence_factor	A multiplier that reflects how likely we believe our results are to be statistically significant.
delta_factor	The difference in the average Critical Metric values between our Variation and the Control Variation. If sign is positive, SiteSpect divides the average difference by 50 to get the delta_factor; if the sign is negative, SiteSpect divides it by 25. This makes negative re-weighting more aggressive than positive re-weighting.
max_reweight_percentage	Setting in the control panel, from 1-100. The default setting is 5. This acts as a multiplier here, and later as a hard maximum for how much a variationgroup can be reweighted.

Keep the following in mind:

Currently, the Auto Optimization feature requires that the campaign is using the default (Original) control group. The Original group must be on, accepting traffic, and set as the campaign control.
For each Variation Group except the Control, SiteSpect calculates the net change by summing the calculated change for each Variation within the Variation Group.
SiteSpect adds the net change to the current Assignment Frequency to get the new Assignment Frequency. SiteSpect does not adjust a Variation Group by more than the Maximum Re-weight Amount setting.
SiteSpect normalizes all the Assignment Frequencies so that the highest weighted Variation Group has an Assignment Frequency of 100. SiteSpect does not adjust the Control Variation Group below the Minimum weight for Controls.
If the Campaign's Optimization is set to Recommendations, SiteSpect shares its recommendations; if it's set to Automatic, it applies them.