Post-Launch: Monitoring the Virtual Assistant

The purpose of this article is to explore some best practices for monitoring and improving virtual assistant performance and responding to user feedback. To help you navigate the article we have broken it down into the following sections:

• What is post-launch monitoring?
• Who has access to complete post-launch monitoring?
• Reviewing Communication Opportunities Interactions
• Reviewing Search Interactions
• Reviewing User Feedback (Optional feature)
  • Negative Feedback with Comments
  • Lowest Performing Responses

What is post-launch monitoring?

Regularly reviewing interactions post-launch and making adjustments based on user feedback is crucial to ensure that your virtual assistant is performing optimally and continuing to meet the needs of your users. We recommend spending 30 - 60 minutes at the end of each month to review your virtual assistant's performance. 

Who has access to complete post-launch monitoring?

The settings needed to complete post-launch monitoring may include the following permissions: 

For more information on user permissions, review the User Roles & Permissions article.

Reviewing Communication Opportunities Interactions

Communication Opportunities are instances in which your virtual assistant was unable to provide any response to a user's input. 

As a result, reviewing and addressing Communication Opportunities interactions is one of the most effective ways you can quickly improve your virtual assistant's performance. Patterns in Communication Opportunities responses may indicate a gap in content about a topic that is important to your users. Adding relevant questions can improve your virtual assistant's performance immediately and reduce the number of Communication Opportunities responses in the future.

1. To review Communication Opportunities interactions, go to the Virtual Assistant>Interactions page.
2. Select the Filter dropdown, and from the Response Type dropdown, select Communication Opportunities. 
   1. If you have more than one page of interactions to review, consider exporting the results as a .csv file that can be further filtered, sorted, and organized in Excel or another spreadsheet software. You can export the interactions by selecting the Export button next to the filter icon.
3. Select the chat icon to view the interaction.
   
4. For a question you would like your virtual assistant to be able to answer if asked moving forward, select Add A Response. 

   
   

   1. In the Response field, craft your response, and select Save.
      

      For more information on crafting your response, review the Creating a Knowledge Base Response article.
      

      
      

   2. After crafting and saving your response, it will be sent to the Moderation Queue for review by Ocelot Staff before being published to your live knowledge base.

If Automatic Content Generation provided the Communication Opportunities response, the Add A Response button will not be available and a new question will need to be created to add it to the knowledge base.

For more information about how to create a new knowledge base question, review the Creating and Editing a Knowledge Base Question article.

Best Practice: You do not need to add every Communication Opportunities interaction to your virtual assistant's knowledge base. Some Communication Opportunities interactions may be overly vague or complex, outside of your contracted AI Department Learning Models, or altogether outside the scope of a virtual assistant designed to answer questions about higher education. 

For more information about what makes an effective question, review the Editing the Knowledge Base Best Practices article.

Regularly monitoring your virtual assistant's Communication Opportunities and adding relevant content will reduce the number of Communication Opportunities interactions in your virtual assistant over time.

If you notice a pattern of questions outside of your contracted departments or interactions in a language other than English in your Communication Opportunities responses, reach out to your CSM. Ocelot offers AI Department Learning Models for many campus offices and translations for several additional languages.

Reviewing Search Interactions

Search responses are instances in which your virtual assistant was unable to provide a response from your knowledge base, and uses your web knowledge source to offer a list of relevant links from your indexed webpages. 

Patterns in search responses can also provide valuable insight into what your users are searching for and whether your virtual assistant is providing helpful suggestions.

1. To review Search interactions, go to the Virtual Assistant>Interactions page.
   
   
2. Select the Filter dropdown, and from the Response Type dropdown, select Search.
    
   1. If you have more than one page of interactions to review, consider exporting the results as a .csv file that can be further filtered, sorted, and organized in Excel or another spreadsheet software. You can export the interactions by selecting the Export button next to the filter icon.
3. Select the chat icon to view the interaction.
   
4. For a question you would like your Virtual Assistant to be able to answer if asked moving forward select Add a response.
   1. In the Response field, craft your response, and select Save.
      
      

      For more information on crafting your response, review the Creating a Knowledge Base Response article.
      

      
      

   2. After crafting and saving your response, it will be sent to the Moderation Queue for review by Ocelot Staff before being published to your live knowledge base.

If Automatic Content Generation provided the search response, the Add A Response button will not be available and a new question will need to be created to add it to the knowledge base.

For more information about how to create a new knowledge base question, review the Creating and Editing a Knowledge Base Question article.

Note: If you are seeing a high-number of responses that resulted in a search response, consider if your institution has made any changes recently to your webpages. 

Administrative-level users can also adjust pages included in web knowledge sources, force run a web knowledge source outside of its regularly-scheduled interval, or adjust when and how often the web knowledge source is run to ensure search results are up-to-date.

For more information about web knowledge source, review the Knowledge Sources article.

Reviewing User Feedback (Optional feature)

User feedback enables virtual assistant users to provide positive (thumbs up) or negative (thumbs down) feedback about general library questions, custom questions, clarifying questions, and integrated questions. 

Negative Feedback with Comments

Users who provide negative feedback to a virtual assistant response also have the ability to submit a comment to provide additional information about their choice. Reviewing negative user feedback can help identify opportunities to improve your virtual assistant's responses to meet your users' needs. 

1. To review user feedback with comments, go to the Virtual Assistant>User Feedback Analytics page.
   
   The analytics will automatically be filtered by the last month, you can select the timeframe dropdown to change the parameters as needed. 
   
   
2. In the Interactions Feedback chart, there will be a percentage for positive and negative feedback, and the number of negative feedback with comments. Under Comments, select View All. This will take you to the pre-filtered interactions page of all negative user feedback that provided comments. 
   
   1. If you have more than one page of interactions to review, consider exporting the results as a .csv file that can be further filtered, sorted, and organized in Excel or another spreadsheet software. You can export the interactions by selecting the Export button next to the filter icon.
      
3. Select the chat icon to view the interaction.
   
4. To modify the response, select Edit the Response button.
5. Edit the response and select Save.
   
   

When reviewing user feedback to make adjustments to virtual assistant content, consider why the user provided negative feedback and what, if any, changes they have suggested.

• Did the response answer the user's question, or point them to the appropriate resource to have it answered?
  
• Did the response provide enough information about a complex process?
• Did the response include enough information for users from different populations, if relevant (e.g., prospective students, current students, alumni, etc.)?
• Is the user asking for additional information or details about a complex task?
  
• Is the user asking for additional contact information or links?

When a user's feedback includes a request for additional details or contact information in a response, these are valuable suggestions that can be used to improve that response for all users.

If Automatic Content Generation created the response, the Edit Response button will not be available and a new question will need to be created to add it to the knowledge base.

For more information about how to create a new knowledge base question, review the Creating and Editing a Knowledge Base Question article.

Best Practice: You may not need to make changes to your virtual assistant's responses based on every piece of user feedback. Some user feedback may be too vague or hyper-specific to be actionable or applicable to all users. Additionally, some user feedback may express disappointment with the response itself, instead of the response's length, level of detail, or written quality.

Lowest Performing Responses

Reviewing and addressing your virtual assistant's lowest-performing responses based on user feedback is another effective way to quickly improve your virtual assistant's performance.

1. To review the lowest-performing responses, go to the Virtual Assistant>User Feedback Analytics page.
2. Scroll to the bottom of the page to find the Lowest Performing Responses chart. Questions are arranged in descending order, with those receiving the greatest amount of negative feedback at the top of the list. 
3. To review feedback for a specific question, select the question from the list.
4. Select the dropdown arrow next to the question, then select Edit or View. 
5. Select the Analytics tab to review all user feedback and comments for that question.
   1. If a question has multiple negative feedback indicators but no comments, you should still review the response for accuracy and add any additional detail that may be helpful to your users.