At times the question arises that the data from a third party analytics system does not align with the user or metric data from SiteSpect
SiteSpect uses a proxy based architecture and tends to be more accurate than tag based systems. The data from different systems can be as close as about 3% and range to 15%. There are a variety of other considerations at play that we will explore below.
The number of users or visits don't align
- Bot identification uses different methods. SiteSpect uses a combination of User Agent Strings and Automatic Robot Detection to pass through those users identified as a bot so they don't end up in your Campaign data. SiteSpect tends to be more stringent at identifying bots.
- Many third-party analytics tools will be configured with extra filters to remove traffic from sources like call centers, internal users, or similar. Additionally, some analytics tools are not integrated into every type of page, such as internal management tools. SiteSpect will see all traffic by default. Check to ensure configuration filters line up between tools.
- Triggers are complex. Variation Triggers determine when an experience is applied and the user is counted in the Campaign. SiteSpect supports complex Triggers that can look at not only a page URL, but also page source contents, cookies, referrers, and other headers for example. Ensure that you are comparing apples to apples.
- OmniChannel, if configured, can identify users from different devices as the same user. Consider applying the OmniChannel segment in the Performance Matrix to see how it changes the User counts.
- Audiences will target user groups and therefore reduce assigned users.
- Percentage of traffic that is routed to SiteSpect can be configured to a value other than 100%. This is a decision that is made during initial configuration. Check with your Optimization Consultant to ensure 100% of traffic is being routed to SiteSpect.
- Traffic Management options such as non-overlay sets to keep assignments between campaigns mutually exclusive and Campaign Frequency to control the percentage of traffic allocated to a Campaign can result in less than 100% of your visits being assigned to a given campaign.
- Visit Expiration settings caused users to be expired from the Campaign.
- Assignment Ceiling has been reached and therefore one or more Campaigns are experiencing traffic starvation.
If Metric hits don't align
- Metrics can be set up to use the same types of Triggers that a Variation uses.
- Metrics will be recorded in your Campaign if the user is counted in their visit.
- Therefore Metrics are subject to the list above that pertains to Campaigns.
- Additionally, Campaigns have the option of either counting all metrics from the start of a visit or only counting metrics that occur after a user is counted.
If the split looks different
SiteSpect randomly splits users in an A/B/n test into one of the Campaign Variation Groups when assignment happens. Users are Assigned to a Campaign once they visit any page that is routed. A user will only be counted in that visit should they Trigger a Variation. As a result, users who do not get counted during their visit will not be included in the Campaign data for that visit.
Other systems can perform the split and assignment when the user matches a certain page. This logic is different from what SiteSpect does by default and could account for a difference in the percentage split for counted users.
SiteSpect has an Audience type called Visited URL. This Audience will delay Assignment until the user matches on a specific Page URL.
Verification and Validation
- WATTS (Web Analytics Tag Transformation and Segmentation) is a way to push Campaign information (most commonly the Assigned and Counted Variation Group IDs) to a third party system such as Google Analytics or Adobe. If this is not set up you might consider doing so as it provides additional visibility. In addition to passing Campaign information SiteSpect can also pass its user ID (SSID / SSGUID) which can be helpful when validating data across tools.
- Validation Campaign. Consider running a simple single Variation Group A/ campaign that counts on some pages well-known to the team that queries that third-party system. Add metrics for those and any other pages or actions that are commonly tracked. Start with a validation campaign that counts all users before trying to validate a campaign with more restricted counting.
- Ancillary cookie. If your site has any sort of cookie per user, SiteSpect can pick that cookie up and include it in the raw data export for investigation purposes and cross identification. This is called the Ancillary cookie (please work with your Optimization Consultant to get this cookie set up).
- Custom Variable Metric. A type of Metric called a Custom Variable can be created to capture any type of data in the source or in a dataLayer for example. This can be used to capture a specific site ID that can help with comparison across tools e.g. user ID, basket ID, order ID, store ID etc (please work with your Optimization Consultant or Solutions Developer to get this set up).
- In general, running one or more validation Campaigns with the mentioned tools and approaches should provide the necessary feedback to validate data alignment exercises.