After pressing "submit for review": a look behind the scenes of the certification team

Anyone who has ever developed a skill for Alexa, whether hobbyist, experienced programmer or professional agency, has already used it: the submit for review button. But what exactly happens in the review? What should you pay attention to and how can you avoid unnecessary resubmissions of your skill?

As a developer, you sometimes ask yourself "Is my skill even being tested by real people?" or "Are the testers using physical Echo devices for certification?" We want to get to the bottom of these questions today. So this is what happens, when you click on the submit for review button:

Right after the click, your skill will be received by the certification team. This team will perform your requested review by testing different categories of your current submission:

• Functional Testing checks the core functionality of the skill and ensures the concept of the skill is technically applicable. For example, the correct connection to the APIs will be checked, but also the example phrases that are used frequently by most of the users. Also the skill description and cards must match the context of the responses, so the experience of potential customers will not be disrupted.
  
  
• During Security Testing, various safety aspects of the skill are carefully examined to ensure that customer data is continuously protected. This includes validating the endpoint, as well as checking the linking of a user account. For example, if a skill requests personal data, the handling of this information must also be explained in an external private policy declaration. Sensitive processes such as money transactions or bookings must also be protected via a voice code as well.
  
  
• Voice Interface and User Experience Testing ensures that language interaction with the skill works smoothly. For example, a skill must always provide relevant answers, even when the user remains silent after a prompt to speak. The same applies to the use of incorrect slot values, as the user may try phrases that are not exactly covered in the interaction model, once the skill is live.
  
  
• In addition, the entirety  of a skill must fit the current Policy Guidelines, before it can become part of the Alexa Skill Store. For certain content, the publication of the skill can be supplemented by an age recommendation. However, if the certification team observe a violation of the guidelines, a content update will be necessary before the skill can go live within the store.

Depending on which countries and languages are selected for publication, different certification teams in several countries around the world work on the review of a single skill.

The testers consciously act like end user and will certify your skill on different device types by using their own voice to simulate a realistic testing environment. If a developer decides to offer display support for his skill as well, screen devices will also be involved during the testing.

A test plan defines the skill-testing framework for different scenarios in order to keep the certification consistent at a global level. The test plan criteria have to be adapted on a regular basis, for example due to API launches or due to new use cases developers implemented. These scenarios have to be evaluated case by case, which might lead to extended processing time within the certification.

As soon as the testers of the individual countries and languages have completed the review, they are consolidated at a central point, cross-checked and sent to the developer.

If the skill passes certification, it will be available in the Alexa Skill Store. If a publication is not possible, required changes will be listed in a feedback email for the developer to support the next submission. Was this feedback helpful or is there something you’re missing? The certification team appreciate your feedback, so ask for your opinion. If you are interested in sharing your experience, just check the bottom of your feedback mail.

Still got questions about your feedback? Our developer advocate team will be happy to help! Feel free to contact us for additional information or other questions regarding your skill. Keep in mind that you should receive at least one piece of feedback from the certification team.

So behind the submit for review button there are not as many secrets as you might think! If you take the time to look at the Submission Checklist, you will also find many of the test scenarios and the required results for a successful certification in advance. If you work through this list conscientiously, the chance to publish your skill in the first submission will increase drastically.


Related Content

Functional Testing for a Custom Skill
Security Testing for an Alexa Skill
Voice Interface and User Experience Testing for a Custom Skill
Policy Guidelines
Certification Requirements - Submission Checklist