Spider creation is the process of building a “spider” that the chatbot uses to search client websites and provide search results in response to user input. Ocelot manually collects a list of a client's main URL paths based on content libraries purchased. We index the main applicable domain(s) and all subsidiary pages in the same path. Pages behind authentication are not automatically spidered but can be. Pages belonging to departments not participating in the chatbot will not be indexed.
The chatbot does not need to be embedded on a webpage to be included in the spider. When a user asks the chatbot a question, an AI-based algorithm is used to identify key intents and entities to match the question with the content on the client's indexed webpages. This spider algorithm operates independently of the AI knowledge base that the chatbot uses to provide direct text and/or video responses to user input.