What is a Spider?
As part of the Chatbot set-up process, Ocelot analyzes your entire website to ensure that your existing pages with applicable content is "spidered" (AKA “indexed” or "crawled") into your Chatbot.
How are Spiders used?
Similar to a textbook, the indexed web pages serve to identify where key information by topic is located on your website.
Spidered website content acts as a type of safety net to provide possible answers to questions when the bot is not able to find a knowledge base match. The AI goes through the following process prior to utilizing spidered content:
- The AI searches for a knowledge base response to the user's question and provides an available match.
- If the bot is not able to provide a response to from the knowledge base (including your custom questions), the AI will attempt to provide suggestions from the knowledge base. The user will be able to choose one of the suggestions to receive a response.
- If the bot is not able to provide suggested knowledge base responses, it will then search your spidered website content to provide links to relevant content on your website.
- If the bot still cannot find related content to provide a response, it will then respond with an IDK ("I Don’t Know") and provide the user with an office contact option.
Who has access to Spiders?
Administrative-level users have access to the Spiders tab on the client admin.
How can Administrative-level users manage Spiders?
Administrative-level users can determine how often Ocelot refreshes your spidered sites so that content adjustments on your website are accounted for in the Chatbot’s search results.
Intervals
Ocelot recommends the interval of Every Other Week (the default) to ensure that content is kept up-to-date without putting a strain on the system. Daily and Weekly intervals should be utilized only when there is an active website redesign in place and content is changing at a fast pace.
Actions
Under the Actions column, there are multiple functions:
- View Content(the eye icon)
Allows you to see all indexed/spidered URLs- Click the View Content icon next to an indexed/spidered URL to view all content that has been crawled on that webpage.
- Force Run Spider
- The Force Run button allows you to immediately crawl your spidered content to refresh any content that may have been updated on the web pages.
- This can take anywhere between 10 minutes to 2 hours to complete.
- Edit Spider
- The Edit Spider button option allows for additional customization of the frequency for which the spider can be run.
- Delete Spider
- The Delete Spider button will stop crawling content for the Chatbot from that point forward.
- Previously crawled content may still be available in the bot. Open a ticket to request a full refresh if you would like to ensure that all previously crawled content is cleared from the bot's memory.
- View Runs
- The View Runs button shows you the history of when the spider has crawled the indexed webpages.
- The Status column will indicate whether there was an update to the content crawled on that web page.
- Under the Actions column, you can view the current content that has been crawled for the web page.
- The View Runs tab can be searched, filtered by Status and Start URL, and exported to a CSV.
- The View Runs button shows you the history of when the spider has crawled the indexed webpages.
If your main URL domains change along the way (i.e. if centennial.edu/admissions changes to admissions.centennial.edu), open a ticket to request a full reset so Ocelot can ensure your spider is properly refreshed.