Testing and Certification
Alexa customers expect a high-quality and consistent experience across all products. The Amazon testing and certification process ensures your product meets customer expectations before you ship.
Product testing is broken into two phases: self-testing (you test your device), and Amazon testing (we test your device). With this guide, you'll learn to test your own device the way Amazon will test it, streamlining the path to certification and reducing time to market. This document also explains the typical Alexa Voice Service (AVS) development lifecycle as it pertains to testing.
- Product testing process
- Launch and post-certification updates (OTA)
- Additional resources
Product testing process
The goal of testing is to validate that your Alexa product meets our many device requirements. Some of these requirements, such as far-field audio performance, can be difficult to test and measure. To help you meet this bar and build the best possible Alexa integration, all devices must go through Amazon’s testing and certification process before being approved for launch. As part of the process, you’ll be executing Amazon-provided self-tests - and ensuring they pass. When you’re ready, and all self-tests pass, you’ll move on to testing by Amazon. The following sections take you through this process in detail.
Understand the requirements
All self-tests tie back to AVS requirements. As you build your device, follow the requirements to ensure a quality Alexa experience:
- Functional Requirements
- Security Requirements
- Program Requirements
- Terms and Agreements
- UX Design Guides
Work with your Amazon representative
Your Amazon point of contact can help guide you through the process throughout several touch-points:
- Creating a detailed timeline outlining the testing process leading up to submission for Amazon testing.
- Understanding which self-test checklists are applicable to your product.
- Gaining access to the self-test checklists.
- How, where, and when to submit for Amazon testing, including the number of physical units to submit.
- Letting you know when your product has passed Amazon testing and reached certification.
- Providing the right next steps and assets for launch.
During self-testing, you'll use a series of downloadable checklists provided by Amazon. Each checklist is designed by our testing team to ensure your product meets all AVS requirements. They cover functional, user experience (UX), security, acoustic, and music self-tests. Ensure your product is well-tested through your own QA before starting self-testing.
If your product supports key features such as Multi-room Music (MRM), is far-field-enabled, or is a unique product type such as a mobile accessory, a different list of self-tests is required. Work with your Amazon point of contact to identify which self-tests apply to your device.
Submit your self-test results
After you've completed the self-test checklists, they must be submitted to Amazon for review. Use the AVS Developer Console to provide your self-test results and other information to Amazon, and to track your progress. You’ll find the latest status and results for each test Amazon conducts.
If you haven’t already, follow the steps to create a product profile.
Submit for Amazon testing
After your self-tests are validated, the testing team will provide you with authorization and instructions to ship your full products to Amazon. Depending on what features your product supports, plan to send us 2-6 devices. Your Amazon contact can help you determine the exact number we'll need.
Once we receive your devices, Amazon will conduct the same tests to ensure your device functions properly, that it provides a quality experience for Alexa customers, and that your product meets our requirements. The Amazon testing process takes approximately 4-11 weeks. If your product isn’t well-tested on your side, or if it doesn’t pass all self-tests prior to submission, you will delay the process on the Amazon side.
Once you’ve submitted your product to Amazon for testing, here's what to expect:
- Acceptance test (BVT): Your product build will be verified using an acceptance test to ensure it meets the basic functional requirements. This test checks a subset of the items in the Functional Self-test Checklist, which you completed during self-testing. We focus on the initial out-of the box (OOBE) experience: Does it turn on? Does it connect to WiFi? Can I register my Alexa account? Can I interact with Alexa?
- Functional and UX testing: If it passes the acceptance test, your product undergoes full testing to ensure it meets the requirements specified in the Amazon Developer Services Agreement, the AVS Functional Requirements and the AVS UX Design Guidelines.
- Security testing: In parallel, arrange for a third-party lab to conduct a security assessment of your device.
- Acoustic testing: If your product is far-field (hands-free) and uses a wake word to initiate interactions with Alexa, acoustic testing will be scheduled and performed at the same time as all other testing.
- Music testing: If your product supports access to third-party music or radio services using the Alexa Service, separate music testing and certification is required for each applicable music service provider (MSP), including Amazon Music, TuneIn, and iHeartRadio.
- Amazon test report: Identified issues and recommendations will be documented and returned to you within an average of 4-11 weeks from the start of Amazon testing.
You can use the Developer Console to regularly check where your product in the process, including the pass/fail status of each test.
Submit for security assessment
Your product is also required to undergo an external Security assessment. You should arrange for your device to be tested by a qualified third-party security lab as early as possible in the development process. The security assessment, just like Amazon testing, can begin as soon as you complete self-testing. Waiting until one is complete to start the other is unnecessary and will slow your time to market. For a list of qualified labs around the world, talk to your Amazon representative.
Once the external security lab completes their assessment, the Amazon Security team will need a few documents in order to complete our certification process:
- The external security assessment report
- An incident response plan
- A software maintenance plan
- A remediation plan
After the Amazon Security team reviews the first three documents, you'll receive a response outlining any critical and high severity vulnerabilities. Then, submit a remediation plan describing how you'll fix all vulnerabilities under active exploitation. Review the full Security Guide for more details.
Leverage your Developer Console product detail page to submit these documents to Amazon before launch.
Launch and post-certification updates (OTA)
If your product passes all Amazon tests, it will be eligible for certification and approval for launch. After launch, any updates or changes to the firmware of your device must be tested by your team before being deployed to products in the field (OTA updates). If significant updates are made, devices that were previously certified by Amazon may be required to be re-submitted for certification, incremental certification, or additional self-testing.
Re-certification or incremental certification by Amazon is required under the following conditions:
- Changes cause an increase in the memory footprint of the device.
- Changes are made to voice processing algorithms such as the audio front-end (AFE) or wake word engine (WWE). Examples include:
- Moving from one WWE to another.
- Moving from one AFE or WWE size to another.
- Moving from one audio player to another.
- Making changes to the AFE effects gain, echo canceling, or tuning.
- Adding new Alexa features and capabilities which were not previously certified.
- Including changes in a major release that is tightly coupled with Alexa interaction.
- Releasing updates in supported regions or languages which were not previously certified.
When re-certification is required, release notes describing the changes must be provided to determine the necessary scope of testing by Amazon. Re-submit your successfully completed Functional Self-test Checklist and Acoustic Self-test Checklist for all supported regions where the OTA update will be released.
The documents listed below provide design, functional, and implementation guidance:
- AVS Functional Requirements
- Self-testing Process
- Audio Hardware Configurations
- UX Design Guides
- Security Requirements
- Marketing and Brand Guidelines