Spider creation is the process of building a “spider” that the chatbot uses to crawl client websites and provide relevant search results in response to a user's input. When Ocelot sets up a chatbot, we create an index of the client's website to identify relevant pages as they pertain to the client's purchased libraries. Similarly to a textbook, the index serves to identify where key information is located by topic.
When a user asks the chatbot a question, an AI-based algorithm is used to identify key intents and entities to match the question with the content on the client's indexed webpages.
This spider operates independently of the AI knowledge base that the chatbot uses to provide direct text and/or video responses to user input. For example, if a user asks "How do I complete the FAFSA?" our AI model works to find an existing knowledge base response to answer the question. Separate from this process but occurring simultaneously, the "spider" algorithm identifies keywords from the user's input and returns links from our index of the client's webpages. The ideal result provides a user with a 1:1 response to their question from the knowledge base plus related links from the client's website to provide additional relevant information.