This section of the user guide is focused on FAQs related to post-launch monitoring of the chatbot. To help you navigate the article we have broken it down into the following sections:
- General
- Analytics
- Are links in the welcome message counted in analytics?
- Are links from the explore bar incorporated into the analytics page?
- When will page-based analytics by office start once URL Matching is completed?
- If my domain changes, how will this affect page-based analytics?
- Why does the Top Topic statistic on the Dashboard not match the number of Interactions when searching for that topic?
- Why is the school's main URL domain (i.e. school.edu) showing as a Top Page on the Dashboard when the school doesn't have the bot embedded on that page?
- Why do the statistics on the Dashboard's Response Types graph not match the number of interactions in the Interactions area?
- Why do my analytics not match my interactions?
- I Don't Know (IDK) Behavior
- Transcripts and Interactions
- Awaiting Review
- User Experience
General
What is the difference between the library and office filters?
Office filter
When an office is set up, libraries are attached to that office. Depending on the setup of the chatbot, multiple offices could have the same libraries, different libraries, or multiple libraries associated with that office. All library content associated with that office will display based on that filter.
Example: The financial aid office is always assigned the financial aid and financial wellness libraries. The Office filter for financial aid will include all questions or analytics for both libraries.
Library filter
The library filter displays the knowledge base questions that have the library selected associated with them.
Can Ocelot help identify students who use the chatbot?
When the inquiry form is enabled and configured, the chatbot presents selected questions to users as soon as the chatbot is opened in order to collect user information.
If the inquiry form is not configured and enabled, Ocelot is only able to identify the IP address associated with a person using the chatbot. Unless the user self-identifies, no identifiable information other than the IP address from which the chatbot was accessed can be provided in the event a client needs to determine a user's identity from an interaction.
Analytics
Are links in the welcome message counted in analytics?
If a link is included in the chatbot welcome message, the link may appear in the Top Clicks and Top Pages in your chatbot analytics:
Are links from the explore bar incorporated into the analytics page?
No. Clicks on links from the explore bar are not currently tracked in our analytics.
When will page-based analytics by office start once URL Matching is completed?
If a client has not yet completed URL matching, their chatbot analytics will include an Office filter and will only be associated with an office based on:
- The office a user selects to speak with a live chat agent.
- The office a user selects to contact because they received an IDK (I Don't Know) response from the chatbot.
- If a user does not engage the chatbot in any of the above ways, the conversation is not associated with an office for analytics.
If a client has completed URL matching, they have access to by-office (i.e. page-based) chatbot analytics. Analytics will be associated with a specific office based on:
- The office the user selects to speak with a live chat agent.
- The office the user selects to contact because they received an IDK response from the chatbot.
- The office the user selects to contact when they ask the chatbot for office contact information.
The webpage URL from which the user engages the chatbot if an office isn't specifically selected based on the above reasons. (This webpage URL should be added in the Look and Feel section under URL Matching. For more information, review the Look & Feel: URL Matching article)
If my domain changes, how will this affect page-based analytics?
Historical page-based (i.e., by-office) chatbot analytics will not be impacted by a domain change.
To ensure that all analytics going forward are not impacted, you will need to update your mapped URL for each office with the new URL.
Why does the Top Topic statistic on the Dashboard not match the number of Interactions when searching for that topic?
Searching for a keyword on the Chatbot Interactions page only returns interactions that contain the exact keyword entered in the Search field.
The Top Topics results displayed on the Chatbot Analytics page have been identified using AI and include all chatbot interactions related to a keyword, including synonyms, misspellings, etc.
Why is the school's main URL domain (i.e. school.edu) showing as a Top Page on the Dashboard when the school doesn't have the bot embedded on that page?
This is due to security settings on these webpages. They have elected to include a header or meta tag on some pages restricting the referrer to "origin," meaning that external applications can only see the domain name rather than the full requested URL.
This setup is typically used for sensitive pages where the URL might contain, for example, a user's search keywords. Some schools include this security tag on every page.
Why do the statistics on the Dashboard's Response Types graph not match the number of interactions in the Interactions area?
Some response types are not tracked in the Response Types graph on the Chatbot Analytics page. The Response Types graph includes the following response types:
- Knowledge base
- Smalltalk
- Suggestion boxes
- Search (links)
- IDK (I Don't Know)
- Integrated
- Other
Other includes responses when the chatbot is unable to determine the user's intent and must ask the user to rephrase their request to clarify and be more specific, or to ask one question at a time. These instances primarily include user inputs that are several sentences or paragraphs long, contain a detailed personal narrative, or are a single word.
Why do my analytics not match my interactions?
If you're experiencing a discrepancy between your analytics and interactions, there are a couple of reasons why this might be happening.
- A library was removed from your chatbot: If you've removed a library or office from your chatbot, the analytics for that library or office will be tracked in your overall analytics. However, you will not be able to filter by that office or library within the Analytics page or Interactions page.
- A conversation did not have a knowledge base interaction: If a conversation did not have a knowledge base interaction, it will not appear in the analytics filtered by that library. This can happen if a user asks a question that is not within your knowledge base, such as smalltalk or an I don't know. Smalltalk and I don't know interactions are not assigned a library.
I Don't Know (IDK) Behavior
What email address will the IDK Contact Form (Intake) Transcript come from, and what is the subject line of the message?
If a client has customized their chatbot to provide users with the option to complete an IDK ("I Don't Know") contact form when the chatbot is unable to provide a response, any completed forms and their accompanying conversation transcripts will be emailed to the IDK Form Contact Email set up on the Behavior Settings page. This email will come from webmaster@ocelotbot.com and will have the subject "Chatbot Submission from 'Student Name' (name@studentemailaddress.com)."
Note: Only individuals with Administrator-level access can configure chatbot behavior settings globally and on a per-office basis.
How can I see the IDK (I Don't Know) contact form transcripts sent to our office?
The IDK (I Don't Know) contact form transcripts can be viewed on the Chatbot Transcripts page in the Admin Portal.
To view the IDK contact forms sent to your office, under the Chatbot section, select Interactions.
In the search bar, enter the Form Submission Success Message, found on the Behaviors Settings page, under IDK Settings.
You can narrow your search by filtering by a date range and/or a specific office.To find the Form Submission Success Message, go to the Behavior Settings page.
Select the Save file icon to save as a CSV file.
To update the I Don't Know (IDK) behavior, review the Behavior Settings: IDK Settings article.
What is the average IDK percent across all Ocelot clients?
Across all clients, 5-7% of all interactions result in an IDK ("I Don't Know") response. This means that 93-95% of chatbot responses come from a match in the knowledge base, suggestions to related knowledge base responses, or links from the website.
Although the chatbot is always learning and you should see a gradual decrease in the frequency of IDK responses over time, you will likely never be able to eliminate all of them; about 5% of questions/comments the chatbot receives are undecipherable or about topics that the chatbot is not designed to answer. Having less than 10% of your chatbot's interactions result in an IDK response is considered successful and is often achievable through some moderate response customization (i.e. completing the Question Templates).
Transcripts and Interactions
What is the compass icon next to a question in the chatbot interactions?
On the Chatbot Interactions page of the admin portal, a compass icon next to a question means that the end user selected that question while using the chatbot's Explore feature.
What is the average number of interactions per conversation that chatbots receive?
The average chatbot conversation consists of 2.9 interactions.
Awaiting Review
Who gets the Awaiting Review email notifications?
Users who have been designated to receive the awaiting review notification will be sent an email 30 days prior to the review date so the response(s) can be approved as-is or changed as needed.
If you would like to set up to receive the Awaiting Review email notifications or change your settings by following the steps outlined below.
- To access the Edit Profile page, select your initials in the top right-hand corner, and select Edit Profile.
- On the Edit Profile page, under the Phone Number section, review the Chatbot Questions Review Opt Out option.
To receive the email notification, make sure the check bot is not selected.
The email notification is not filtered by office permissions. You may receive an email and have no questions that need to be reviewed by your area.
User Experience
What are the differences between the grey suggestions and suggestions in the Explore feature?
Suggestion boxes are the AI model's efforts to find a direct response to a user's input from the questions in a client's knowledge base. Relevant suggestions in response to user input means that the chatbot is functioning correctly.
The Explore feature provides added value by supplementing these suggestions with further options for users to delve into.
How does the chatbot know what links to present to a user?
Spider creation is the process of building a “spider” that the chatbot uses to crawl client webpages and provide search results if a knowledge base answer or video is not available in response to a user’s question. When Ocelot sets up a chatbot, we create a custom spider for the client to identify relevant webpages as they pertain to the chatbot libraries purchased.
When there isn't a direct match to a user question in our knowledge base, the AI runs an algorithm against the client's spidered website content. It then runs each potential match against the AI model and returns the top match, or gives the user a few relevant options.
The crawler can be run at an interval chosen by the client to ensure website content pulled into the chatbot is current. Unless otherwise selected by a client, the spider runs every other week.
For more information on adjusting your Spider settings, review the Navigating the Admin: Spiders article.
What is Simplified Chinese? Is this Mandarin?
Mandarin and Cantonese are two dialects of Chinese; "dialect" refers only to spoken language. The chatbot communicates in Simplified Chinese, which is written.
Because Simplified Chinese refers only to the written word while Mandarin and Cantonese refer to spoken dialects, document translation--like a chatbot interaction--cannot be conducted in Mandarin or Cantonese, and Simplified Chinese is not used in spoken conversation.
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article