Understanding the Skill Intent Confidence Dashboard

To ensure a positive customer experience, you should regularly check how your Skill interaction model is performing. It would be very difficult to understand what is and isn’t working with your Alexa Skills without solid analytics. With the new Intent Confidence Dashboard, you can better understand how users are interacting with your skill and where they might be having issues.

The Intent Confidence Dashboard gives you the ability to analyze your skill’s quality as well as providing new opportunities to optimize its performance. For example, you can detect anomalies where customer requests are mapped to the wrong intent with high confidence, resulting in undesired skill responses. You can also identify requests that resolve to an intent with low confidence, resulting in Alexa re-prompting the customer for a response or identifying missing use cases that you may want to include in your skill. In addition, you can detect popular slot values in different locales.

The skill performance section will provide you with a collection of metrics to track overall skill accuracy. To help you get the most out of the new Intents Confidence dashboard, this post walks through some of the questions you may have around this section of the analytics.

The new dashboard indicates how your skill interaction model is performing. On this page you can see the number of utterances resolved to an intent with high, medium and low confidence:

These metrics will allow you to gain insight into your skill’s performance and possible obstacles to the users, which can help you improve the skill experience. By checking the Medium and Low confidence intents you will know where to update your skill interaction model for a more qualitative skill.

As you can see in the dashboard above the 93,767 represents the number of intents and entities get successfully resolved along with high confidence. 41 represents the number utterances that had low confidence and because of that, Alexa could not match to any intent

Similarly, when you see a higher rate of failures in a specific Entity Resolutions slot, you can add more slot values or synonyms and update your live skill within minutes using Instant Publish.

A Low Intent Confidence might result in a poor customer experience for the end user. For example, if a user says something that Alexa cannot match to an intent, it will trigger the AMAZON.Fallbackintent, and you will then be able to ask the user to repeat or provide more information. As a developer you may want to update the interaction model to include common out of scope utterances or update the handler to be more helpful for users.

You should:

1. Use the fallback intent to handle unexpected utterances, or when a customer says something that doesn’t map to any intents or slot in your skill.

2. Review unresolved utterances and map them to an intent or slot in your model if they are reasonable requests. Find those unresolved utterances in your skill’s intent history that could be low confidence intent. Review the intent history.

You can navigate to your skill’s Intent History under the Build tab in the Alexa Developer Console. The example below shows the type of data you might see for a custom skill, that represents the number of utterances that were said by multiple unique customers:

At the same time please keep in mind that the Intent History is only displayed if your skill has 10 or more active users in a day in a specific locale.

3. Review your lambda function code to learn how Alexa is responding.

You can customize your report by selecting the Custom range button and choosing the time frame that you want to see. You can also use the Time Interval controls to change the date range (today, yesterday, last 7 days, last 30 days, or by selecting a custom date range). Using the Aggregation Period drop-down list, you can change how the data is grouped (every 15 minutes, hourly, daily, or weekly).

You can also consult skill intent confidence metrics by skill stages: Live and Development. Development can include skills tested in beta. Skills must be submitted and certified to become Live.

Lastly, you can select the Grid view and start to play with your skill data in order to see the detailed aggregated results by hour, daily, weekly and by choosing different time intervals: