Sample Size Calculator
Simple Sample Size Calculator
SiteSpect has a sample size calculator for estimating the required sample size and thus run time for any experiment. The SiteSpect sample size calculator can be found here. The calculator has two options to select from and the primary KPI for the experiment will determine which is used, either uniques or totals.
Glossary of Terms
Alpha (α): The maximum probability of Type I error, or the probability of rejecting a true null hypothesis. In the context of sample size calculation, alpha represents the maximum risk of incorrectly concluding that a statistically significant difference exists when there is no real difference. A common value for alpha is 0.05, which means that there is a 5% chance of Type I error. This is often reflected as a signficance level of 95%.
Beta (β): The maximum probability of Type II error, or the probability of failing to reject a false null hypothesis. In the context of sample size calculation, beta represents the maximum risk of missing a statistically significant difference when it exists. A common value for beta is 0.20, which means that there is a 20% chance of Type II error.
Baseline Mean: The average value of the outcome variable in the population, or the value that the sample mean is being compared to. The baseline mean is used as a reference point to calculate the sample size required to detect a statistically significant difference.
Targeted MDE (Minimum Detectable Effect): The smallest difference between the sample mean and the baseline mean that the researcher wants to be able to detect with a certain level of confidence. The targeted MDE is used to calculate the sample size required to detect a statistically significant difference of a certain size.
Standard Deviation: A measure of the amount of variation or dispersion in the population. The standard deviation is used to calculate the sample size required to detect a statistically significant difference, taking into account the amount of variation in the population.
Expected Traffic: This is the daily expected traffic to your variation group for the experiment location. It should take into account traffic only to those locations and consider any audience includes/excludes to ensure an accurate input.
Recommendations for usage
A good way to think about the inputs for the
SiteSpect sample size calculator is to break down the fields into those that require a speciifc value that is accrurate and sourced from a
SiteSpect baseline campaign or your web analytics data vs those you can modify or use as levers to impact your run time.
The remaining fields of
targted MDE %, Alpha & Beta are those you can increase or decrease to impact your sample size and run time requirements. For example, if you want to run to 90% significance then you could set the Alpha value to 0.1 which is accepting a little more risk of a false positive (1 in 10 vs 0.5 in 10). Similarly, if you are more bullish on your experiment concept you may wish to set our MDE higher at 5-10% rather than 2-3%. MDE and sample size are inversely linked meaning the smaller or more "sensitive" the MDE, the larger the sample size needed to reach statistical significance.
A helpful way to think about setting values for targted MDE % is the impact of trying to detect a small pebble being thrown into a pond which would take some time because of the general noise of the water. Now consider this vs a large boulder being dropped into the same pond. Knowing what you believe about the scale of the change and its potential to impact your KPI, you can consider what is the largest and smallest possible impact % MDE for that input. With selecting this input (as with the Alpha) there is always a trade off. Higher MDE is good for maintaining a smaller sample size and shorter run time. But if you become too bullish you may overstate the potential and miss out on a smaller signal.
Another area for consideration is the location of your experiment and its relationship and distance to your KPI. In some cases while certain experiment traffic may be considered lower due to its site location e.g. the checkout the baseline % is higher and thus the MDE can be set at a lower/reasonable level. Thus the metric KPI you select is another lever you can consider in this calculation. The run time for a sitewide experiment on orders is likely to require a lot of time / very large sample. So you might consider a KPI that is closer to the changes which can help to manage the sample more realistically.
Last but by no means least, an important additional lever to consider is the design itself. Be much bolder with the changes (throw a bigger boulder!) as this will likely provide a larger signal and potential MDE to detect, thus reducing sample size.
Further reading
