Alexa Built-in Testing and Certification Process
Alexa customers expect a high-quality and consistent experience across all products. The Amazon testing and certification process ensures your product meets user expectations before you ship.
Product testing has two phases:
- Self-testing – You test your own device. See Self-testing Process.
- Amazon certification testing – Amazon tests your device for certification.
Testing your own device the same way that Amazon tests streamlines your path to certification and reduces development time. This page explains the testing process in detail for Alexa Built-in devices that implement the Alexa Voice Service (AVS).
- Product testing process
- Launch and post-certification Over-the-Air (OTA) updates
- Additional resources
Product testing process
The testing process validates that your Alexa product meets Amazon-specified device requirements. Some requirements, such as far-field audio performance, can be difficult to test and measure. To help you meet Amazon standards and build the best possible Alexa integration, all devices must go through Amazon’s testing and certification process before receiving approval for launch. As part of the process, you’ll be executing Amazon-provided self-tests, and ensuring that your device passes. When you’re ready, and all self-tests pass, you’ll move on to testing by Amazon.
Understand the requirements
All self-tests tie back to AVS requirements. As you build your device, follow the requirements to provide a quality Alexa experience:
- Functional Requirements
- Security Requirements
- Program Requirements
- Terms and Agreements
- UX Design Guides
Work with your Amazon representative
Communicate with your Amazon point of contact to make sure that your testing process goes smoothly:
- Create a detailed timeline outlining the testing process leading up to submission for Amazon testing.
- Understand which self-test checklists are applicable to your product.
- Access the self-test checklists.
- Learn how, where, and when to submit for Amazon testing, including the number of physical units to submit.
- Work together to resolve issues during certification.
- Wait for notification from your representative that your product has passed Amazon testing and reached certification.
- Communicate about the appropriate next steps and receive your assets for launch.
During self-testing, use the downloadable checklists provided by Amazon. The Amazon testing team designed each checklist to verify that your product meets all AVS requirements. The checklists cover functional, user experience (UX), security, acoustic, and music self-tests. Thoroughly test your product through your own QA before starting self-testing.
Products implementing features such as Multi-room Music (MRM) support, far-field support, or Bluetooth support require testing against a different set of checklists. Work with your Amazon point of contact to identify which self-tests apply to your device.
Amazon requires music feature certification for devices implementing Alexa Music before launching the device to the public. Alexa Music certification ensures that the device adequately supports music partners such as Amazon Music, iHeartRadio, and SiriusXM. To help test your device against the Music Certification checklist, Amazon offers a Music Validation Tool (MVT) for developers.
Submit your self-test results
After completing the self-test checklists, submit them to Amazon for review. Use the AVS developer console to provide your self-test results and to track your progress. The console also provides the latest status and results for each test that Amazon conducts on your device.
If you haven’t already, follow the steps to create a product profile.
Submit your device to Amazon for testing
After validating your self-tests, the testing team authorizes you to proceed and provides instructions to ship your products to Amazon. The number of devices required for submission varies, depending on which features the device supports. Most Alexa Built-in devices require you to send in 2-6 devices for testing. Ask your Amazon contact for the required number of devices.
After Amazon receives your devices, the testing team executes tests to verify device functionality, product requirements, and that the device provides a quality experience for Alexa customers. The Amazon testing process typically takes 4-11 weeks. If you didn't thoroughly test your own device, or if the device didn't pass all self-tests before submission, you'll delay the testing process for Amazon.
After submitting your product to Amazon for testing, the Amazon testing team conducts device certification testing in the following order and compiles a feedback report:
- Acceptance test (BVT): Amazon verifies your product build with an acceptance test to make sure that it meets the basic functional requirements. The Acceptance test checks a subset of the items in the Functional Self-test Checklist, which you completed during self-testing. Checklist requirements focus on the initial out-of the box (OOBE) experience: Does it turn on? Does it connect to WiFi? Can I register my Alexa account? Can I interact with Alexa?
- Functional and UX testing: If it passes the acceptance test, your product undergoes full testing to verify that it meets the requirements specified in the Amazon Developer Services Agreement, the AVS Functional Requirements and the AVS UX Design Guidelines.
- Security testing: In parallel, arrange for a third-party lab to conduct a security assessment of your device.
- Acoustic testing: If your product is hands-free and uses a wake word to invoke interactions with Alexa, the testing team schedules and performs acoustic testing at the same time as all other testing.
- Music testing: If your product supports access to third-party music or radio services through AVS, Amazon conducts separate music testing and certification for each applicable MSP, such as Amazon Music, TuneIn, and iHeartRadio.
- Amazon test report: Identifies and documents issues and recommendations for your device. Expect to receive the report between 4-11 weeks from the start of Amazon testing.
Use the AVS developer console to track where your product in the process, including the pass/fail status of each test. Additionally, watch for email notifications from Amazon for status updates.
Submit for security assessment
Your product is also required to undergo an external Security assessment. Arrange for your a qualified third-party security lab to test your device as soon possible in the development process. Like Amazon testing, begin the security assessment as soon as you complete self-testing. Waiting until one is complete to start the other is unnecessary and might delay your product launch. For a list of qualified labs around the world, log in the the AVS developer console, and download the AVS Security External Labs document from your whitelisted Amazon developer account.
After the external security lab completes their assessment, the Amazon Security team requires the following documents to complete the certification process:
- External security assessment report
- Incident response plan
- Software maintenance plan
- Remediation plan
After the Amazon Security team reviews the first three documents, you'll receive a report outlining any critical and high severity vulnerabilities. In response, submit a remediation plan describing how you plan to fix all vulnerabilities under active exploitation. For more details, log in the the AVS developer console, and download the AVS Device security requirements _ intake form.xlsx from your whitelisted Amazon developer account.
Leverage your developer console product detail page to submit these documents to Amazon before launch.
Launch and post-certification Over-the-Air (OTA) updates
After your product passes all Amazon tests, it becomes eligible for certification and approval for launch. Test any post-launch changes to the firmware of your device before deploying any updates to products in the field through Over-the-Air (OTA) updates. If you make significant device updates, Amazon might require that you re-submit your device for re-certification, incremental certification, or more self-testing.
Amazon requires re-certification or incremental certification under the following conditions:
- Increasing in the memory footprint of the device.
- Altering the voice processing algorithms such as the audio front-end (AFE) or wake word engine (WWE). Some examples include the following update types:
- Migrating from one WWE to another.
- Changing the AFE or WWE size.
- Migrating from one audio player to another.
- Changing the AFE details, echo canceling, or tuning.
- Implementing uncertified new Alexa features and capabilities.
- Coupling changes in a major release with Alexa interaction.
- Releasing updates for uncertified regions or languages.
If Amazon requests re-certification, submit a set of release notes describing your changes to determine the necessary scope of testing by Amazon. Re-submit your completed Functional Self-test Checklist and Acoustic Self-test Checklist for all supported regions where you plan to deploy the OTA update.
The following documents provide design, functional, and implementation guidance for AVS: