As part of the Chatbot set-up process, Ocelot takes the main office homepages purchased in your contract and analyzes your entire website to ensure all applicable related content is spidered (or “indexed”) into your Chatbot. Similar to a textbook, the index serves to identify where key information by topic is located.  


Spidered website content acts as a safety net to provide responses when the bot does not have a response available in the knowledge base.

  • If the bot is not able to provide a response to a user from the knowledge base (including your custom questions), the AI will attempt to provide suggestions from the knowledge base. The user will be able to choose one of the suggestions to receive a response.

  • If the bot is not able to provide suggested knowledge base responses, it will then search your spidered website content to provide a relevant link to a page on your website.

  • If the bot still cannot find related content to provide a response, it will then respond with an IDK ("I Don’t Know") and invite the user to contact the office.


Administrative-Level users have access to the Spiders tab on the client admin.


The Spiders section in the admin will allow you to determine how often Ocelot refreshes your spidered sites so that content adjustments on your website are accounted for in the Chatbot’s search results




Ocelot recommends the interval of Every Other Week to ensure that content is kept up-to-date without putting a strain on the system. Daily and Weekly intervals should be utilized only when there is an active website redesign in place and content is changing at a fast pace.


Under the Actions column, there are multiple functions

  • View Content (the eye icon)
    • The view content icon allows you to see all indexed/spidered URLs
    • Click the View Content icon next to an indexed/spidered URL to view all content that has been crawled on that webpage. 

  • Force Run Spider
    • The Force Run button lets you immediately crawl your spidered content to refresh any content that may have been updated on the web pages.
      • It should take 10 minutes to run but could take up to a couple of hours if the queue is busy.
  • Edit Spider
    • The Edit Spider button option allows for additional customization of the frequency the spider can be run. 

  • Delete Spider
    • The Delete Spider button will stop crawling content for the Chatbot but previously crawled content might still be available in the bot.
  • View Runs
    • The View Runs button shows you the history of when the spider has crawled your indexed webpages. 
      • The Status column will indicate whether there was an update to the content crawled on that web page. 
      • Under the Actions column, you can view the current content that has been crawled for the web page.
      • The View Runs tab can be searched, filtered by Status and Start URL, and exported to a CSV.



If your main URL domains change along the way (i.e. if centennial.edu/admissions changes to admissions.centennial.edu), notify your Client Success Manager so we can ensure your spider is properly refreshed.