Site
      Back to home
      1. Help Center
      2. Site

      The SiteSpect Engine API robots, bots, & crawlers management

      Engine API Implementations

      There are two methods for utilizing the SiteSpect Engine API which will impact how you can approach bot management. The SiteSpect Engine API can be implemented server-side or client-side from the browser. If your implementation uses the SiteSpect reverse-proxy please reference this.

       

      Engine API on the server-side only:

        • Does not use Automatic Robot Detection (ARD). 

        • Does not use passthrough list. This can be enabled if a valid browser User Agent is passed from the app. Please contact sitespect@helpdesk.com to discuss this.

        • Do not use the IAB/ABC International Spiders and Bots List within SiteSpect. Common scripting languages where user-agents are not customized will be considered robots. e.g. curl and python are some common user-agents that will be unintentionally flagged a robots.

        • It is recommended you only send SiteSpect Engine API requests for non bot traffic. One method to achieve this would be through the use of a CDN.

      • Engine API client-side / in the browser:

        • Does not use Automatic Robot Detection (ARD).

        • Passthrough list can be enabled if required. Please contact sitespect@helpdesk.com to discuss this.

        • IAB/ABC International Spiders and Bots List can provide bot, crawlers and browser detection. The level of IAB detection required will depend on your SiteSpect configuration so please speak to your SiteSpect Consultant, Account Manager or email helpdesk@sitespect.com to discuss enabling this.

        • It is recommended you only send SiteSpect Engine API requests for non bot traffic. One method to achieve this would be through the use of a CDN.