Analyze Your Skill Metrics
To help you understand how users interact with your Alexa skills, Amazon provides tools to generate usage metric reports for each skill. Usage metrics include the number of unique customers, the percentage of users who invoked a particular intent interaction path, and more. To view your skill metrics reports, visit the Analytics page in the Alexa developer console. Alternatively, you can access skill metrics with the Metrics API.
Access the Alexa skill Analytics dashboard
To access the Analytics dashboard in the Alexa developer console and learn how to generate reports, see View Skill Metrics.
Available skill metric categories
Metrics are available for different categories of skills, such as custom or smart home skills. (No metrics are available for meeting skills.) Most metrics are available for skills at both the development and live stages. All metrics data excludes invocations from the Test your skill page in the developer console.
The following table provides an overview of the categories of metrics available for your skills:
Metric | Description | Skill types |
---|---|---|
Actions |
Unique users per action, total actions, and total utterances per action. |
Smart Home |
Customers |
Definition of this metric depends on the type of skill.
|
Custom |
Intents |
Unique users per intent, total utterances per intent, total intents, and failed intents. |
Custom |
Interaction Path |
Paths users take when interacting with your skill. For example, view the percentage of users who invoked a particular intent after the |
Custom |
Play Queues |
Number of play queues, average play queues per customer, total play queue duration, average play queue duration in hours, play queue duration in hours per customer. |
Music, Radio, and Podcast |
Playback |
Percentage of play queues that lasted 30 seconds or longer. |
Music, Radio, and Podcast |
Plays |
Total number of times that a user played the skill content. |
Flash Briefing |
Retention (live skills only) |
Usage of your skill over time by groups of users or cohorts. View the number or percentage of users who returned to your skill over a 12-week period. |
Custom |
Sessions |
Total sessions, successful session types (sessions that didn't end due to an error), average sessions per user. Includes a breakdown of successful, failed, and no-response sessions as a percentage of total sessions. |
Custom |
Top Media Content |
Most-played content items. |
Music, Radio, and Podcast |
Utterances |
Metrics for utterances depend on the skill category.
|
Custom |
Summary of skill metrics
The Summary section of the Analytics page displays an overview of the metrics available for the skill. For each metric, you have options to view infographic-style charts, grids, or other visualizations. Available metric data export formats are PNG or JPEG images for infographics, or CSV format for raw data.
![Analytics page](https://m.media-amazon.com/images/G/01/mobile-apps/dex/ask-devconsole/measure-page._TTH_.png)
Skill outage notification
On the Analytics page, the developer console displays a high-severity event banner when Alexa detects that the skill is experiencing an outage. For details about the outage, click Learn More.
Skill quality coach
The Skill quality coach analyzes your skill against common attributes of successful skills, and then generates a quality score and improvement recommendations. Use these recommendations to build high-quality Alexa skills that can increase customer engagement.
At this time, the skill quality coach is available for published custom skills in the United States (English) with enough monthly active users to measure customer feedback accurately.
Quality score
The Skill Quality Score metric is a composite score, measured on a scale from 0 – 5, that indicates the completeness of the skill and strength of customer feedback.
The quality score is based on the following skill attributes:
- Rich multimodal and voice responses
- Voice interaction model coverage and accuracy
- Fresh, engaging, and personalized content
- Skill endpoint health, such as reliability and latency
The quality score evaluates customer feedback based on the following input:
- Customer stickiness, including number of engaged, repeat, and lapsed customers
- Customer feedback, including ratings and reviews
Together these metrics produce a quality score between 0 and 5. The following table shows the relationship between the metrics and the score.
Low customer feedback | High customer feedback | |
---|---|---|
Less common skill attributes |
0 – 1 |
1 – 3 |
More common skill attributes |
1 – 3 |
3 – 5 |
Recommended insights
The skill quality coach provides Recommended Insights to improve your quality score. Each recommendation is based on your skill type and attributes and includes estimated effort to implement the improvement. To improve your skill quality, implement the recommendations and republish your skill.
The skill quality coach provides the recommendations based on the following input:
- Automated evaluation of your skill against high-quality design features for similar skills.
- Skill insights team that curates recommendations for select skills based on thorough evaluation of the skill.
To help improve the insights evaluation, you can provide feedback on the recommendations. Provide thumbs up if you agree and thumbs down if you disagree with a recommendation.
Skill activation and token retrieval metrics
For skills that use account linking, the Analytics page has an Account Linking section that displays metrics related to skill activation and token retrieval. These metrics are available for live skills.
To understand these metrics, first learn about account linking initiations and completions. For details, see Understand Account Linking.
The following table shows the metrics that are available under Skill Activation. These metrics show data up to two days before the viewing time.
Metric | Description | Skill types |
---|---|---|
Total Enablements |
Definition of this metric depends on the type of skill.
Although some users might enable your skill, they might never launch that skill. Users who launch a skill count as unique user. This behavior can explain a difference between the number of times users enable a skill and the number of unique users. |
Custom (without account linking) |
Total Account Linking Events |
Total number of account linking initiations and completions. |
Custom (with account linking) |
Account Linking Completion Rate |
Percentage of successful account links. The percentage is the total number of account linking completions divided by the total number of initiations. |
Custom (with account linking) |
Token retrieval metrics
The following table shows the metrics that are available under Token Retrieval. These metrics appear in near real-time (within a few minutes of being emitted).
Metric | Description | Skill types |
---|---|---|
Access Token Requests: Success Rate | Percentage of successful access token requests from the Alexa service to your skill's authorization server. For details, see Overview of the authorization code grant flow. | Custom (with account linking - authorization code grant) Music, radio, or podcast (with account linking) Smart home |
Refreshing Access Token Requests: Success Rate | Percentage of successful access token refresh requests from the Alexa service to your skill's authorization server. For details, see Overview of the authorization code grant flow. | Custom (with account linking - authorization code grant) Music, radio, or podcast (with account linking) Smart home |
Access Token Requests: Error Responses | Number of error responses to access token requests from the Alexa service to the authorization server for the skill. For details on errors that come from OAuth 2.0, see Error Response in the OAuth 2.0 authentication framework specification. If the Alexa service receives an OAuth 2.0 error that it doesn't recognize, it displays it as an UNKNOWN_ERROR_CODE. A TRANSIENT_ERROR indicates that the Alexa service wasn't able to communicate with the partner service. For other error situations, Alexa returns a null error. |
Custom (with account linking - authorization code grant) Music, radio, or podcast (with account linking) Smart home |
Refreshing Access Token Requests: Error Responses | Number of error responses to access token refresh requests from the Alexa service to your skill's authorization server. For details on errors that come from OAuth 2.0, see Error Response in the OAuth 2.0 authentication framework specification. If the Alexa service receives an OAuth 2.0 error that it doesn't recognize, it displays it as an UNKNOWN_ERROR_CODE. A TRANSIENT_ERROR indicates that the Alexa service wasn't able to communicate with the partner service. For other error situations, Alexa returns a null error. |
Custom (with account linking - authorization code grant) Music, radio, or podcast (with account linking) Smart home |
Operational metrics
Use operational metrics to see how your smart home and video skills perform. On the Analytics page, in the Operational Metrics section, the Discovery, Capability Directives, and Reporting State pages display the latency and success rate metrics for device discovery, device control, query state requests, and proactive change reports, respectively. For each metric, graphics, such as charts, grids, and other visualizations, are available. Export graphics in PNG or JPG file formats, or download raw data in CSV file format.
The reports for operational metrics include the following analytics:
-
Accuracy rate – Accuracy rate indicates the percentage of time that the response to a
ReportState
directive matched a previously sentChangeReport
event.To maintain Works with Alexa certification, your product should achieve 97 percent change reporting accuracy. For details, see Recommended performance targets.
- Latency – After Alexa sends a request to your skill, latency is the time in milliseconds until Alexa receives a response from your skill. Amazon measures latency with two types of statistics:
- Percentile (P) – Percentile, such as P99, P90, and P50, indicates how a value compares to others within the same period. For example, P90 is the 90th percentile and means that 90 percent of the data within the period is lower than this value and 10 percent of the data is higher than this value. Average indicates the sum of values divided by the total number of data points.
- Trimmed Mean (TM) – Trimmed mean is the average of a set of measurements, after including or discarding values indicated by number or range, such as TM95 and TM95:99. TM metrics more accurately reflect latency at volume than single percentile-based metrics. The following examples show TM metrics:
- TM95, with no specified ending range, is the average latency after throwing away the highest five percent of latency data. TM95 latency is an indicator of overall latency performance as it discards the outlier high latency values.
- TM95:99, with a range of 95–99 percent, focuses on a segment of the highest latency values collected, averaging latency values that fall in the specified range. Use this data to identify edge cases of high latency and understand peak latency as compared to a more inclusive metric, such as TM95.
Use the Choose Data menu (three vertical dots in a circle icon) in the latency graphs to choose different percentile and trimmed mean statistics or the average. A single graph displays multiple statistics options.
To maintain Works with Alexa certification, for capability directive responses, your product should have a maximum latency of 1000 milliseconds for P90 and 800 milliseconds for P50. For details, see Recommended performance targets.
-
Request volume – Both latency and success rate graphs show request volume on the second Y-Axis. Use the Choose Data menu (three vertical dots in a circle icon) on the graphs to toggle the request volume on and off.
- Success rate – Success rate is the total number of successful responses Alexa receives from your skill divided by the total number of requests Alexa sends to your skill.
In the Operational Metrics section, you can view the following reports:
-
Discovery – Alexa sends a discovery request to your skill to discover your customer's smart devices. The Discovery page shows the latency and success rate graphs for all discovery requests to your skill. For more details about discovery, see Alexa Discovery interface.
- Capability Directives – When a user asks Alexa to control their smart device, Alexa sends a directive request to your skill to trigger a capability, for example, turning on a light. For more details about capability directives, see the documentation for each interface that your skill supports. On this page, you can download logs for troubleshooting.
The Capability Directives page shows the following tabs:- Overview – Shows a summary view of latency and success rate for all capabilities. In addition, the tab shows a summary of latency and success rate issues and the number of utterances affected by country and device type. To view details about the issues, see Download logs for operational issues.
- Latency – Shows latency data for the aggregate of all capability interfaces and for each capability interface supported by your skill. Each per-capability latency graph shows the aggregate across all directives and the breakdown per directive for all directives supported by the capability.
- Success Rate – Shows success rate data for the aggregate of all capability interfaces and for each capability interface supported by your skill. Each per-capability success-rate graph shows the aggregate across all directives and the breakdown per directive for all directives supported by the capability.
-
Reporting State – Alexa sends a report state directive to your skill to request the state of an endpoint. The Reporting State page shows the latency and success rate graphs for all the report state requests to your skill. For more details about report state, see State Reporting for a Smart Home Skill.
- Change Report – When the state of a device changes for any reason, the device reports that change to Alexa in a
ChangeReport
event. Then, Alexa provides the status change to the user in the Alexa app. The Change Report page shows Accuracy Rate metrics that reflect the percentage of time that theChangeReport
event matched a previousStateReport
response. Accuracy Rate metrics include both manual and Alexa-directed state changes, such as when a user physically turns on a light and when a user asks Alexa to turn on the light. For more details about change reporting, see Report state in a ChangeReport.
The following image shows example latency and success rate graphs for the last 30 days, aggregated daily. The latency graph includes selected percentile and trimmed mean data. Latency graphs are available on the Discovery, Capability Directives, and Reporting State pages.
![Dashboard shows summary Discovery operational metrics.](https://m.media-amazon.com/images/G/01/mobile-apps/dex/ask-smart-home-docs/smart-home-analytics/TrimmedMean._TTH_.png)
Skill performance metrics
Use skill performance metrics to view how well your skill is functioning.
- Endpoint Latency – Use Endpoint Latency to see how responsive your skill is to user utterances. Skills with low endpoint latency have slow responses and cause friction for users.
- Endpoint Response – The Endpoint Response metric shows the number of successful and failed responses your skill has over time. Skills that have no response confuse users and lower user satisfaction with your skill. If users aren't responding to your skill, your endpoint-response metrics reflect a low score.
- Intent Confidence – The Intent Confidence metric shows how your voice interaction model is performing. The confidence setting options are high, medium, or low. A low confidence request from a user triggers a re-prompt instead of sending the utterance to your skill as an intent request, resulting in a poor user experience.
-
Estimated System Errors – The Estimated System Errors metric estimates the potential success of a conversation turn based on system signals from ASR/NLU and context. Use this metric to review utterances that frequently cause failed interactions and to view aggregate skill improvements over time so that you can modify your skill to further improve accuracy.
Note: The Estimated System Errors metric is available for high-traffic skills in en_US, en-GB, and de-DE.
Monetization metrics
Alexa has three payment models for paid skills and custom skills with in-skill products:
- One-time purchase – A single transaction purchase that unlocks access to the skill itself or to premium content within the skill. For example, access to a paid skill or to a new level within a game. A one-time purchase doesn't expire.
- Subscription – A purchase that offers access to your skill or premium-skill content for a period of time, charged on a recurring basis until the customer cancels the subscription. If you offer a free trial, after the trial period ends, access converts to a paid subscription.
- Consumables – (For in-skill products only) A purchase that unlocks access to content, such as in-game currency or extra lives. The customer consumes and depletes the content, and might re-purchase it repeatedly.
The following table shows the metrics that are available under Monetization.
Metric | Description | Applicable types |
---|---|---|
Total Purchases |
Total number of offers accepted for all purchases within a skill. For consumables, Total Purchases includes repeat purchases by the same user. |
One-time purchases and consumables |
Purchases |
Number of offers accepted for each one-time purchase or consumable within a skill. For consumables, Purchases includes repeat purchases. |
One-time purchases and consumables |
Offer Impressions |
Total number of times users heard the purchase prompt from Amazon, describing a skill or product and its price, and decided whether to buy. Offer Impressions includes both free-trial and paid offers. For consumables, this metric includes both the first-time purchase and repeat purchases of the same item. Offer Impressions doesn't include messages provided within the upsell directive. |
One-time purchases, consumables, subscriptions |
Offer to Purchase Conversion |
The percentage of offer impressions that resulted in a purchase. |
One-time purchases and consumables |
Active Subscriptions |
Total number of active subscriptions for your skill, including paid subscriptions with future expiration dates. |
Subscriptions |
Trial to Paid Conversion Rate |
Percentage of trial subscriptions that became paid subscriptions. |
Subscriptions |
Subscriptions Started |
Total number of new subscriptions. |
Subscriptions |
Subscriptions Ended |
The total number of trial subscriptions that a user canceled or paid subscriptions that a user canceled. If a user cancels a paid subscription, the subscription remains active until it expires at the end of the billing period for the user. |
Subscriptions |
Churn |
Percentage of active paid subscriptions that expired or that a user canceled. |
Subscriptions |
Offer Conversion Rate |
Percentage of offer impressions that resulted in the user signing up for a new trial or paid subscription. |
Subscriptions |
Upsell to Offer Conversion |
(For in-skill products only) The percentage of upsell impressions that resulted in a purchase. Upsell to Offer Conversion is available for the upsell directive only. |
One-time purchases, consumables, subscriptions |
Upsell to Purchase Conversion |
(For in-skill products only) The percentage of upsell impressions that resulted in a purchase. This metric is available for the upsell directive only. |
One-time purchases, consumables, subscriptions |
Alexa Shopping Actions metrics
Alexa has the following categories of Shopping Actions for custom skills:
- Add-to-Cart action – The customer requests that the skill add the product to their Amazon Shopping Cart. Later, the customer visits Amazon.com to make the purchase.
- Add-to-Wishlist action – The customer requests that the skill add the product to their Amazon Wish List.
- Buy action – The customer purchases the requested product during the skill session.
- Recommend-products action – The customer agrees to learn more about products that your skill recommends.
Metric | Description |
---|---|
Add to Cart recommendations |
Total number of product recommendations made within the skill. |
Add to Cart conversions |
Number of product recommendations that the customer added to their Cart. |
Add to Wishlist recommendations |
Total number of product recommendations made within the skill. |
Add to Wishlist conversions |
Number of product recommendations that the customer added to their Wish List. |
Buy Action recommendations |
Total number of product purchase recommendations made within the skill. |
Buy Action conversions |
Number of product recommendations that the customer purchased. |
Number of product recommendations |
Total number of product recommendations made within the skill by using the Recommend-products action. |
Recommend products Add-to-Cart conversions |
Number of product recommendations that the customer added to their Cart. |
Recommend products Buy conversions |
Number of product recommendations that the customer purchased. |
Customer friction metrics
Use customer friction metrics to see how users interact with your skill. Alexa uses various user interaction patterns and other contextual signals to predict if a user experienced friction when interacting with your skill.
-
Customer friction summary – Use the Customer friction summary dashboard to see relative trends for your skill plus a list of frequent utterances that Alexa has identified as causing potential customer friction.
-
Barge-ins – Use the Barge-ins metric to see how often user interrupt an ongoing skill response. High barge-in rates could have causes, such as long skill responses, unexpected up-selling options, or not allowing enough time for a user to respond to a prompt.
-
Terminations – Use the Terminations metric to see the number of times users prematurely end a skill session. Premature termination causes include a user making a request that's out of scope for the skill, an incorrect slot recognition, or an incorrect use of Name Free Invocation.
-
Unhandled Requests – Use the Unhandled Requests metric to see the number of times your skill is unable to handle user requests. New user requests that might be relevant for your skill can contribute to these unhandled requests.
Related topics
- View Skill Metrics
- Interpret and Troubleshoot Skill Performance Metrics
- Download Message IDs for Troubleshooting
- Skill Metrics FAQ
Last updated: Mar 26, 2024