Alexa Skill Analytics & Maintenance - Alexa Champion Ralf Eggert

I’m very proud to present a guest blog post by Alexa Champion Ralf Eggert. He is currently founder and CEO of Travello, an officially recognised Alexa partner agency for Alexa Skill design and development. Since the beginning of 2017 Ralf also personally designs and develops skills some of which have become very popular. Fun fact: he powers the backend of his skills through a custom PHP framework he developed himself, called phlexa.

By Ralf Eggert, Alexa Champion

It is now relatively easy to create a new Alexa Skill. There is ample support for the development, the design and the marketing of Alexa Skills. Developers can follow YouTube videos or Twitch streams to learn and fine tune their Skills even more. However, what happens to an Alexa Skill after it has been released? What options and tools should a developer us to secure the smooth operation while at the same time continuously improving the Alexa Skill? This blog post hopes to shed some light on how to go about this by covering…

At Travello we found that almost every Alexa Skill goes through the same broad project phases. They are as follows:

1.                Kick-off

2.                Prototyping

3.                Conception

4.                Design

5.                Implementation

6.                Testing

7.                Maintenance

I won’t go into the individual phases (1-6) in detail at this point. In principle, it doesn't matter whether someone has identified more, fewer or different project phases in their workstream as different teams have different ways of working. What is critical however, is what happens after the publication of an Alexa Skill: the so-called maintenance phase.

The maintenance phase for any Alexa Skill project will be the longest phase in its life cycle. Hardly any skill is “finished” after its first publication. The odds are it will need to be patched, tweaked, and improved on again. This maintenance phase is actually a broad meta-phase in which the previous phases such as conception, design, implementation and testing are regularly run through for new features or improvements. We call this phase the Continuous Maintenance Phase. In the following I explain the different tools and possibilities to continuously improve a published Alexa Skill in this phase.

The metrics that can be found for every Alexa Skill in the Alexa Developer Console under the tab »Analytics« represent a valuable source for the analysis of your Alexa Skill. Here you will find a lot of data and analytical insights to ensure proper visibility on the success of your skill and to identify possible technical issues. There are definitely some parts which are worth a regular look.

The Summary tab should be viewed daily or at least once a week (preferably on the same day of the week). The default setting here is seven days and that should be more than sufficient if you consult this section regular regularly. In this way, it is easier to identify rising or falling trends. One word of advice (based on experience): you shouldn't panic if your skill drops in a week or if it suddenly goes through the roof. Weekly fluctuations are quite normal. It is more about long-term trends. You can see below an example screenshot of my Summary tab:         

• The Customers tab we use to identify, among other things, the days of the week or times of day with the highest or lowest use. Some Alexa Skills may be used most frequently during the weekend or in the evening, others may be a certain day of the week (e.g. trash collection days). Why is this important? For example, if we implement features that could potentially disrupt the live Skill experience, we should pay close attention to avoid our “hot zones” or peak times of usage. This can also now be controlled by deciding when to publish the latest version manually after a successful certification.         
• In the Sessions tab, it is often possible to identify problematic situations. If an above-average number of sessions abort or expire at certain periods, this can indicate a technical problem. At Travello we were once able to identify a backup process that was running on high load and increased the response times a lot. This caused the backend responses to the Alexa Voice Service to take too long and consequently causing the answers fail for the users. A reset of this backup process solved the problem. We wouldn’t have caught this without consulting this tab.
• The Intent view is a good way to identify the most used and less used intents. You should continuously improve the most used intents for example by controlling for more edge cases while you can also likewise remove less used intents to make your application more robust. Often, simpler is better (and easier to maintain!)         

• The Interaction Path tab is super cool! It is most useful in the case of planned dialogues that should lead your users through several intents, it can be used to check whether this really works or whether the users somehow lose the thread. In one of our Alexa Skills, we were able to recognise that a certain intention was called up to 4x in succession, although only a single call makes sense.   

• Last but not least, if your Alexa Skills are in it for the long term, you should definitely consult the Retention tab. I recommend to monitor this section at least every couple weeks after every major change. It can be used to answer various questions. How many users come back after a week and how many after ten weeks? Can you see progress over a longer period of time? How are your changes affecting your retention?

This tool is a data mine, so check it regularly and most importantly: act on any issues you spot!

I’ll cover the Intent History separately from the Analytics section as I don’t consider it to be distinct from the other metrics previously discussed and it is found on a completely different part of the Developer Console: in the » Build « tab.

When you first look at the Intent History of a well-performing Alexa Skill, you might feel overwhelmed by the shear amount of data. To get started, I recommend the following step-by-step procedure for analysing your Alexa Skills using the Intent History:

• You don't see any data yet? Then the Alexa Skill is either too fresh or still has too few users. In that case, just continue with the analysis of your next Alexa Skill and try to get more users!
• If there is data, look at the » Unresolved Utterances « section from the last 30 days. These are constantly updated, so you should check them regularly. The unresolved utterances give you a sense of what users are saying that you didn’t previously control for. 
• Use the export functionality to extract the data for an in-depth analysis in your preferred spreadsheet software.        
• Start with the utterances marked as "low confidence". There is a very good chance that you will find utterances that are mapping to the wrong intent. These must definitely be corrected, preferably daily or at least weekly. See this screenshot example below:

• The utterances with »medium confidence« are usually a little more difficult to solve. For example, there could be more than one solution or intent to react on this utterance. However, these should also be checked and corrected regularly. 
• When working on any utterance issues, keep in mind that this changes your interaction model. Existing intents are assigned new utterances. As a result, your changes will only take effect with a new certification or an update of the live skill. 
• If you maintain your interaction model outside of the developer console (for example, through the ASK CLI), you should absolutely make sure that your changes in the interaction model are also transferred to your local repository or your own backend. For many Alexa Skills we have set up our own backend with a custom CMS. In it, we also maintain the interaction model so we always sync changes every time.
• Incidentally, the Intent History can also be a great source for new feature ideas or intents. For example, if you receive many utterances for playing music, you could respond to these requests by implementing a dedicated intent to provide a graceful error handling (e.g. “you are still in Skill X, if you want to play music just say exit”). You could also respond to insults from users in a humorous or de-escalating manner (this has worked great for our Skills!)        

The most important thing about the Intent History is that you check it regularly. It gives you the awareness you need to solve problems better and even get ideas for new features.

The reviews in the Alexa Skill catalogue can be a great source of good news, as well as bad. They are an essential feedback tool for you and can also contain information about problems and tips for new features. Here, too, we have developed some best practices:

• Positive written reviews are first and foremost nice to read and give you the confirmation that you have done something right. However, it is difficult to extract approaches to improve an Alexa Skill based on these positive ratings. But use it for your own motivation. Occasionally, fans of a Skill will also suggest new features in their positive review.
• Even 5-star ratings without text are also a great motivational boost, but usually don't really help with the analysis and improvement process of your Skill because they were received without justification and void of any details. However, good job!         
• Purely negative reviews, which are also destructive or unjustified, usually do not help either. Some negative reviews do not deal directly with the Alexa Skill. Maybe the user was just having a bad day or their coffee was cold. Don't waste too much energy on these reviews but if you feel they are unjustified you can also contact support.
• Constructive negative reviews are the most helpful. They hurt psychologically, but they are great sources of feedback on how you can improve your Alexa Skill. Take any constructive criticism seriously and to try to solve the problem in a timely manner. 
• It can be helpful to respond to both negative and positive reviews, even if the chances of the user reading it and updating it themselves are very small. However, since other users can also read your replies, it strengthens the confidence that an Alexa Skill is currently being looked after, that the developer cares about the user experience, and can be a source of additional support as customers scroll through (similar to a Q&A).

In summary, cherish any positive reviews but embrace the (constructive) negative ones.

You shouldn’t blindly adopt any new feature that the Alexa Team provides immediately. Having said that, you should keep yourself informed about innovations as some can be relevant for your particular use case and could open up new experiences that were not previously possible. This can be done via this Alexa Developer Blog or via the Developer Newsletter. Another great source is the Feature Updates section.

Sometimes, a new feature can instantly solve an older problem that you had to find a workaround for. Other times, parked ideas from the past that you couldn't implement with the feature-set at the time suddenly become unblocked. So definitely keep yourself informed about these innovations and make your own roadmap for the future.

I hope this post was able to shed some light on how you can continuously improve and expand your Alexa Skills. Check the metrics regularly, clean up problematic utterances in the intent history, don't be afraid of negative reviews and keep up to date with what's new. Cheers!