Spider creation is the process of building a “spider” that the chatbot uses to search client websites and provide search results if a knowledge base answer or video is not available in response to a user’s question.
Spiders Behind Authentication
- Our spiders are built to crawl public web pages. We do not place extra security or encryption on the data that our spider collects.
- Our spiders can crawl pages behind authentication. But the content crawled will be treated the same as publicly-available content. For example, the spider will collect data such as keywords and phrases, and make portions of that index visible without authentication. The collection of keywords and phrases is done automatically and does not go through a secondary filtering process, nor does Ocelot perform any manual quality control (QC).
When directing Ocelot to spider certain webpages, especially those behind authentication, clients should ensure the page does not contain sensitive or personal information. The contents of these pages will be crawled and indexed per the above process. If a user clicks on a link that requires a login, they will still be required to log in.