Alexa Built-in Testing and Certification Process
Alexa customers expect a high-quality and consistent experience across all products. The Amazon testing and certification process ensures your product meets user expectations before you ship.
Product testing has two phases:
- Self-testing – You test your own device. For more details, see Self-testing Process.
- Amazon certification testing – Amazon or an Authorized Third-Party Lab tests your device for certification.
Testing your own device the same way that Amazon tests streamlines your path to certification and reduces development time. Continue reading to learn about the testing process in detail for Alexa Built-in devices that implement the Alexa Voice Service (AVS).
Product testing process
The testing process validates that your Alexa product meets Amazon-specified device requirements. Some requirements, such as far-field audio performance, can be difficult to test and measure. To help you meet Amazon standards and build the best possible Alexa integration, all devices must go through the Amazon testing and certification process before receiving approval for launch. As part of the process, run the Amazon-provided self-tests, and make sure that your device passes. When you're ready, and all self-tests pass, proceed to have your device tested by Amazon.
Understand the requirements
All self-tests tie back to AVS requirements. As you build your device, follow the requirements to provide a quality Alexa experience:
- Functional Requirements
- Security Requirements
- Program Requirements
- Terms and Agreements
- UX Design Guides
Work with your Amazon representative
Communicate with your Amazon point of contact to make sure that your testing process goes smoothly:
- Create a detailed timeline outlining the testing process leading up to submission for certification testing.
- Understand which self-test checklists are applicable to your product.
- Access the self-test checklists.
- Learn how, where, and when to submit devices for certification testing, including the number of physical units to submit.
- Work together to resolve issues during certification.
- Wait for notification from your representative that your device has passed certification testing.
- Communicate about the appropriate next steps and receive your assets for launch.
During self-testing, use the downloadable checklists provided by Amazon. The Amazon testing team designed each checklist to verify that your product meets all AVS requirements. The checklists cover functional, user experience (UX), security, acoustic, and music self-tests. Thoroughly test your product through your own QA before starting self-testing.
Products implementing features, such as Multi-room Music (MRM) support, far-field support, or Bluetooth support require testing against a different set of checklists. Work with your Amazon point of contact to identify which self-tests apply to your device.
Amazon requires music feature certification for devices implementing Alexa Music before launching the device to the public. Alexa Music certification ensures that the device adequately supports music partners, such as Amazon Music, iHeartRadio, and SiriusXM. To help test your device against the Music Certification checklist, Amazon offers a Music Validation Tool (MVT) for developers.
Submit your self-test results
After completing the self-test checklists, submit your test results either to Amazon through the developer console or to an Authorized Test Lab for review.
If you haven't already, follow the steps to create a product profile.
Submit your device for certification testing
After validating your self-tests, the testing team notifies you and provides instructions to submit your product for certification testing. The number of devices required for submission varies, depending on which features the device supports. Most Alexa Built-in devices require two to six devices for certification testing. Ask your Amazon contact for the required number of devices.
You have the option of submitting your devices either directly to Amazon, or to speed up the certification process by submitting your devices to an Authorized Test Lab. For a list of Authorized Test Lab options, see Authorized Test Labs.
After Amazon receives your devices, the testing team runs tests to verify device functionality, product requirements, and that the device provides a quality experience for Alexa customers. The Amazon testing process typically takes four to eleven weeks. If you didn't thoroughly test your own device, or if the device didn't pass all self-tests before submission, you could delay the Amazon testing process.
After submitting your product to Amazon for testing, the Amazon testing team conducts device certification testing in the following order and compiles a feedback report:
- Security testing: In parallel with the rest of certification testing, arrange for one of the Authorized Third-Party Labs to conduct a security assessment of your device.
- Acceptance test (BVT): Amazon verifies your device build with an acceptance test to make sure that it meets the basic functional requirements. The Acceptance test checks a subset of the items in the Functional Self-test Checklist, which you completed during self-testing. Checklist requirements focus on the initial out-of the box (OOBE) experience: Does the device turn on? Does the device connect to Wi-Fi? Can users register their Alexa account? Can users interact with Alexa?
- Functional and UX testing: If the device passes the acceptance test, your product undergoes full testing to verify that it meets the requirements specified in the Amazon Developer Services Agreement, the AVS Functional Requirements and the AVS UX Design Guidelines.
- Acoustic testing: If your device is hands-free and uses a wake word to invoke interactions with Alexa, the testing team schedules and performs acoustic testing at the same time as all other testing.
- Music testing: If your device supports access to external music or radio services through AVS, Amazon conducts separate music testing and certification for each applicable MSP, such as Amazon Music, TuneIn, and iHeartRadio.
- Amazon test report: Identifies and documents issues and recommendations for your device. Expect to receive the report between four to eleven weeks from the start of Amazon testing.
Use the AVS developer console to track where your product in the process, including the pass/fail status of each test. For status updates, watch for email notifications from Amazon.
Submit your device for a security assessment
Your must submit your device for an external security assessment. Arrange for a qualified security lab to test your device at the beginning of your product development process. As with Amazon testing, begin the security assessment as soon as you complete self-testing so as not to delay your product launch. For a global list of qualified labs, see Authorized Security Labs.
After the authorized lab completes their assessment, the Amazon Security team requires the following documents to complete the certification process:
- External security assessment report
- Incident response plan
- Software maintenance plan
- Remediation plan
After the Amazon Security team reviews the first three documents, they send you a report outlining any critical and high severity vulnerabilities. In response, submit a remediation plan describing how you plan to fix all vulnerabilities under active exploitation. For more details, log in to the AVS developer console, and download the AVS Device security requirements _ intake form.xlsx from your approved Amazon developer account.
Leverage your developer console product detail page to submit these documents to Amazon before launch.
Launch and post-certification Over-the-Air (OTA) updates
After your product passes all Amazon tests, it becomes eligible for certification and approval for launch. Test any post-launch changes to the firmware of your device before deploying any updates to products in the field through Over-the-Air (OTA) updates. If you make significant device updates, Amazon might require that you re-submit your device for recertification, incremental certification, or more self-testing.
Amazon requires recertification or incremental certification under the following conditions:
- Increasing in the memory footprint of the device.
- Altering the voice processing algorithms, such as the audio front-end (AFE) or wake word engine (WWE). Some examples include the following update types:
- Migrating from one WWE to another.
- Changing the AFE or WWE size.
- Migrating from one audio player to another.
- Changing the AFE details, echo canceling, or tuning.
- Implementing uncertified new Alexa features and capabilities.
- Coupling changes in a major release with Alexa interaction.
- Releasing updates for uncertified regions or languages.
If Amazon requests recertification, submit a set of release notes describing your changes to determine the necessary scope of testing by Amazon. Re-submit your completed Functional Self-test Checklist and Acoustic Self-test Checklist for all supported regions where you plan to deploy the OTA update.
If your product is implementing a major software update, such as an operating system upgrade or integrating Alexa Communications, you must submit a security reassessment report. Work with one of the Authorized Security Labs to retest your product against current security requirements and then provide a reassessment report. Ask your Amazon point of contact or email email@example.com for more details about the security reassessment process. For a list of test lab options, see Authorized Security Labs.
The following documents provide design, functional, and implementation guidance for AVS: