In web experimentation, every metric has four ways to position its measurement. For that metric you may consider it as a unique boolean operator that is either true or false, or as a frequency. Boolean interpretations of metrics answer questions of the form “what proportion of”, whereas frequency interpretations of a metric answer questions of the form “at what rate.”
You may also consider the metric in isolation of a singular visit, or across all visits.
Taken together, this can be easily understood in the context of a 2x2 grid, here below with the example of engagement with “new arrivals”.
|
Engagement w/ New Arrivals |
Uniques |
Totals |
|
Visits |
Was ‘new arrivals’ viewed in this visit, y/n? |
How many times was ‘new arrivals’ viewed in this visit? |
|
User |
Did this user see ‘new arrivals’ at least once across all of their visits? |
How many times does each user see ‘new arrivals’ across all of their visits? |
For every experiment, the level of granularity that is best to consider when developing a hypothesis statement is the level of one of the 4 boxes above containing italic text. In general they answer questions differently:
- Uniques, by visit → For each visit, was the metric satisfied at least once?
- Totals, by visit → For each visit, how many times is it satisfied?
- Uniques, by user → For each user, did they satisfy it ever?
- Totals, by user → For each user, how many times has it been satisfied, ever?
Each of these answers different kinds of analytical questions, and has different forms of answers:
- Uniques, by visit → What proportion of visits include this action/outcome?
- Totals, by visit → In each visit, how often is the metric satisfied?
- Uniques, by user → What proportion of our users are doing this action/outcome?
- Totals, by user → For each user, how many times have they done this action overall?
It is clear how each of these frames addresses different analytical questions, and therefore is relevant for different hypotheses.