SiteSpect identifies requests from robots and crawlers using Automatic Robot Detection (ARD) and excludes them from any Campaign data.
ARD is a SiteSpect feature that provides very accurate detection of robots, crawlers, and other User-Agents that do not explicitly identify themselves as such. These are often referred to as cloaked robots since they masquerade as official browsers, and are not filtered by SiteSpect's explicit Pass-Through settings.
To use the ARD feature, which requires system administrator privileges:
- Select Site, Configuration, Site Settings.
- Select the User Tracking tab and scroll to Automatic Robot Detection settings.
Robot Detection Location
- Prepend to top of page – This is the recommended (and default) setting, and accurately filters robots while still capturing even those users who quickly enter/exit a site (i.e., a bounce). The test code is inserted just after the opening of the <head> tag where available, or at the top of the page otherwise.
- Append to bottom of page – This accurately filters robots and most rapid bounces where a human user clicks to a site, views only one page, then quickly presses their "back button" to return to the referring page (typically a search engine). You may want to use this setting if you do not want rapid bounces counted towards your Campaign data. The test code is inserted just before the end of the tag where available, or at the end of the page otherwise.
- Append to absolute bottom of page – Similar to the prior option, but only attempts to append the test code to the end of the page. This option is recommended only for Sites where the prior two options are problematic.
- Off – ARD is disabled, and only those robots that are explicitly filtered by pass-through settings or HTTP Request Exclusions are caught. When ARD is disabled, its settings lower on this page are not visible.
Robot Detection Method
Select the method you want to use to inject SiteSpect's Automatic Robot Detection onto the page:
- new Image() – This is a non-blocking method for injecting code onto the page using a new HTMLImageElement instance. While it does not block the drawing of the page or subsequent HTTP requests, it can block/delay the load event on the window.
- document.write() – We do not recommend this choice, since it is a blocking injection and will be deprecated soon.
Disable Robot Detection Header Name and Value