What are AA, Validation or Insight campaigns, what are best practices to consider when using them, and how can I create one?
An A/A, AA or validation campaign involves serving a campaign on your site using SiteSpect that makes no physical or visible change to the end user. These campaigns can be used to establish baseline averages and totals for metrics, to learn about your users’ behaviors, or to validate the accuracy of the tool. Tests of this kind are referred to as A/A or AA because the variation group is no different than the control. You may even choose to run only a control to simplify data collection and analysis.
Designing an A/A campaign to validate the tool & statistical model
In experimentation, sampling always introduces some difference as no two sample groups are perfectly alike. What is important is for that difference to be small enough to not confound the ability to correctly interpret the test results.
When using the A/A campaign with SiteSpect for this kind of validation, it is best to treat them as any other experiment in that you should choose a metric as a KPI for the test, and then pre-calculate the minimum sample size needed to prove (or disprove) a hypothesis regarding the observed effect size of that KPI. Unlike a standard A/B test, however, the goal is not to demonstrate an effect is present, but to show that the effect found by the A/A test is within an acceptable tolerance for the purpose of testing. In fact, to ‘prove’ that there is no difference whatsoever between the two variation groups of an A/A (an effect size of 0%), the minimum sample size needed would be infinite!
If the A/A test is allowed to run indefinitely, the model can become overly sensitive to even tiny differences, and may show a statistically significant result that is not of practical importance. Therefore, the recommended steps for planning and executing an A/A test to validate the tool are focused around validating that the margin of error of the tool and data are within an acceptable tolerance. Here are steps to accomplish this:
- Choose a KPI that is central to your testing goals, or is the KPI for your upcoming experiment. Be sure to choose a binomial metric such as unique conversions.
- Determine a tolerance (otherwise known as margin-of-error) that is acceptable for the testing purposes. A starting point may be around one-half of the standard-deviation of the baseline KPI.
- Using the margin of error as you would an effect-size for an A/B test, calculate the minimum sample-size needed to run the validation using a sample-size calculator.
- Once the sample-size is determined, set up and run the test for the predetermined amount of time.
- When the test concludes, examine the test KPI results and confirm that the observed effect is within the tolerance specified in step 2.
If the observed effect is outside of the range of tolerance deemed acceptable for your experimentation purposes, it may be necessary to investigate your site traffic for problematic user agents, or to make use of SiteSpect’s bot detection solutions.
It is possible to very quickly deploy these types of campaigns in SiteSpect using the AB build flow. For step by step directions, see below.
Creating an A/A, AA, validation or insights campaign is quick and easy:
- Create a New AB Campaign.
- Name the campaign clearly so it is obvious to SiteSpect users that it is an A/A, AA, validation or insights campaign.
- In the General section select the Overlay Set for campaign.
- Go to the advanced settings and set the priority to 1. This will ensure the campaign recieves the maximum traffic if you are running more then the sites concurrent assignment limit. This is the maximum number of campaigfns a user can have for their session.
- In the Variation section, populate the campaign triggers you need to specify where the campaign should deploy. If this is global for every page then a simple Page Source contains: </head> would be sufficient. Should you wish to focus the campaign in a more specific area of your site then alternative triggers would be needed depending on the site's URL structure. e.g. URL contains: /cart for the cart page or /product for product pages.
- You can choose to leave the Find & Replace blank to ensure no change is deployed against the triggers. However, if you have APIs or other assests routing through SiteSpect you may match on these unintentionally. To avoid this simply match your page source trigger in both the find and replace. In this example above we used </head>.
- If running an A/ campaign without an additional variation group, switch [OFF] the Variation Group leaving just the Control enabled.
- Review the Metric section to ensure the metrics you wish to track in the campaign have been included.
- Assess the Audience section to make sure the campaign is including and excluding the desired users types.
- Save the campaign setup.
QA the campaign using Preview to confirm that the campaign's setup for triggers, metrics and audiences meet your requirements before activating.