The SiteSpect Engine API robots, bots, & crawlers management

Engine API Implementations

There are two methods for utilizing the SiteSpect Engine API which will impact how you can approach bot management. The SiteSpect Engine API can be implemented server-side or client-side from the browser. If your implementation uses the SiteSpect reverse-proxy please reference this.

 

Engine API on the server-side only:

    • Does not use Automatic Robot Detection (ARD). 

    • Do not use the IAB/ABC International Spiders and Bots List within SiteSpect. Common scripting languages where user-agents are not customized will be considered robots. e.g. curl and python are some common user-agents that will be unintentionally flagged a robots.

    • It is recommended you only send SiteSpect Engine API requests for non bot traffic. One method to achieve this would be through the use of a CDN.

  • Engine API client-side / in the browser:

    • Does not use Automatic Robot Detection (ARD).

    • IAB/ABC International Spiders and Bots List can provide bot, crawlers and browser detection. The level of IAB detection required will depend on your SiteSpect configuration so please speak to your SiteSpect Consultant, Account Manager or email helpdesk@sitespect.com to discuss enabling this.

    • It is recommended you only send SiteSpect Engine API requests for non bot traffic. One method to achieve this would be through the use of a CDN.