AVS Testing and Certification Process
Alexa customers expect a high-quality and consistent experience across all products. The Amazon testing and certification process ensures your product meets user expectations before you ship.
Product testing has two phases:
- Self-testing – You test your own device. For more details, see Self-testing Process.
- Amazon certification testing – Amazon or an Authorized Third-Party Lab tests your device for certification.
Testing your own device the same way that Amazon tests streamlines your path to certification and reduces development time. Continue reading to learn about the testing process in detail for Alexa Built-in devices that implement the Alexa Voice Service (AVS).
Product testing process
The testing process validates that your Alexa product meets Amazon-specified device requirements. Some requirements, such as far-field audio performance, can be difficult to test and measure. To help you meet Amazon standards and build the best possible Alexa integration, all devices must go through the Amazon testing and certification process before receiving approval for launch. As part of the process, run the Amazon-provided self-tests, and make sure that your device passes. When you're ready, and all self-tests pass, proceed to have your device tested by Amazon.
Understand the requirements
All self-tests tie back to AVS requirements. As you build your device, follow the requirements to provide a quality Alexa experience:
- Functional Requirements
- Security Requirements
- Program Requirements
- Terms and Agreements
- UX Design Guides
Work with your Amazon representative
Communicate with your Amazon point of contact to make sure that your testing process goes smoothly:
- Create a detailed timeline outlining the testing process leading up to submission for certification testing.
- Understand which self-test checklists are applicable to your product.
- Access the self-test checklists.
- Learn how, where, and when to submit devices for certification testing, including the number of physical units to submit.
- Work together to resolve issues during certification.
- Wait for notification from your representative that your device has passed certification testing.
- Communicate about the appropriate next steps and receive your assets for launch.
Prepare for the security assessment
Before you start your product development process, review the AVS security requirements intake form. You can find this form in the AVS developer console under the developer resources tab. This intake form provides you with a full list of the applicable security requirements that you must meet, based on the device category of your product.
Before you begin building your product, consider the following:
- Choose hardware that satisfies the hardware security requirements listen in the form. For example, you should choose a qualified System-on-Chip (SoC) that relies on hardware-based security capabilities, a hardware-based microphone design, and an operating system that has Long Term Support (LTS).
Review your hardware choices with the AVS security team to confirm your architecture is compliant with the AVS security requirements.Note: Make sure you choose a secure System-on-Chip (SoC) that's PSA Level 1 compliant. You can review the sub-requirements (1.16.x) to verify that the SoC you choose has the correct hardware security capabilities. Check with AVS Security to confirm if the SoC is acceptable.
After you confirm that your device meets all the requirements listed in the security intake form, you must submit your device for testing with an authorized security lab. To avoid product launch delays, you should begin your security assessment as soon as you complete your AVS certification self-testing. Make sure you have a stable build before you submit your device to the security lab for testing.
To complete the security assessment, you must submit the following documents to the Amazon security team:
- AVS security requirement intake form – Lists each of the applicable security requirements you must meet to launch your product with AVS.
- Open source binary scan report (security requirement 1.13) – A scan of your full filesystem (firmware image). The AVS team uses this scan to identify any possible vulnerabilities in open source libraries you might be using. With the firmware image, also provide a top level PDF summary and a full .CSV export listing any vulnerabilities.
- Incident response plan (security requirement 1.6) – A security response plan that describes how your company will proceed if a security incident arises. Details must include when your company will communicate with Amazon if an incident occurs and an estimated timelines for remediation. Make sure you include security contact information for your company and timelines for when you will notify Amazon (or other partners).
- Software maintenance plan (security requirement 1.4) – Defines how your software updates are created and distributed if vulnerabilities are identified. Read through the sub-requirements (1.4.x) to ensure your plan covers at least those topics.
- Microphone schematic design document (security requirement 1.15) – A schematic document that provides details on the required microphone designs elements on your device. This includes including a hardware-based mechanism to control the on/off state of your microphones/cameras and a dedicated status indicator displaying these states.
- External security assessment report (security requirement 1.7) – The third-party security assessment report provided by an AVS-approved security lab, which provides an in-depth security review of your device. For more details about security vendors, see authorized security labs.
- Remediation plan (security requirement 1.8) – A report of known exploitable security vulnerabilities on your device and a plan to fix them. After you complete the security assessment from the security lab, they send you a report outlining security vulnerabilities. In response, you must submit a remediation plan to AVS Security. In this response, describe if the vulnerabilities are applicable, how you plan to fix vulnerabilities, and when you will fix the vulnerabilities.
To submit these documents to Amazon, use the developer console product detail page.
During self-testing, use the downloadable checklists provided by Amazon. The Amazon testing team designed each checklist to verify that your product meets all AVS requirements. The checklists cover functional, user experience (UX), security, acoustic, and music self-tests. Thoroughly test your product through your own QA before starting self-testing.
Products implementing features, such as Multi-room Music (MRM) support, far-field support, or Bluetooth support require testing against a different set of checklists. Work with your Amazon point of contact to identify which self-tests apply to your device.
Amazon requires music feature certification for devices implementing Alexa Music before launching the device to the public. Alexa Music certification ensures that the device adequately supports music partners, such as Amazon Music, iHeartRadio, and SiriusXM. To help test your device against the Music Certification checklist, Amazon offers a Music Validation Tool (MVT) for developers.
Submit your self-test results
After completing the self-test checklists, submit your test results either to Amazon through the developer console or to an Authorized Test Lab for review.
If you haven't already, follow the steps to create a product profile.
Submit your device for certification testing
After validating your self-tests, the testing team notifies you and provides instructions to submit your product for certification testing. The number of devices required for submission varies, depending on which features the device supports. Most Alexa Built-in devices require two to six devices for certification testing. Ask your Amazon contact for the required number of devices.
You have the option of submitting your devices either directly to Amazon, or to speed up the certification process by submitting your devices to an Authorized Test Lab. For a list of Authorized Test Lab options, see Authorized Test Labs.
After Amazon receives your devices, the testing team runs tests to verify device functionality, product requirements, and that the device provides a quality experience for Alexa customers. The Amazon testing process typically takes four to eleven weeks. If you didn't thoroughly test your own device, or if the device didn't pass all self-tests before submission, you could delay the Amazon testing process.
After submitting your product to Amazon for testing, the Amazon testing team conducts device certification testing in the following order and compiles a feedback report:
- Security testing: In parallel with the rest of certification testing, arrange for one of the Authorized Security Labs to conduct a security assessment of your device.
- Acceptance test (BVT): Amazon verifies your device build with an acceptance test to make sure that it meets the basic functional requirements. The Acceptance test checks a subset of the items in the Functional Self-test Checklist, which you completed during self-testing. These checklist requirements primarily focus on your initial out-of the box (OOBE) experience, such as does your device turn on? Does your device connect to Wi-Fi? Can users register their Alexa account? Can users interact with Alexa?
- Functional and UX testing: If your device passes the acceptance test, your product undergoes full testing to verify that it meets the requirements specified in the Amazon Developer Services Agreement, the AVS Functional Requirements and the AVS UX Design Guidelines.
- Acoustic testing: If your device is hands-free and uses a wake word to invoke interactions with Alexa, the testing team schedules and performs acoustic testing at the same time as all other testing.
- Music testing: If your device supports access to external music or radio services through AVS, Amazon conducts separate music testing and certification for each applicable MSP, such as Amazon Music, TuneIn, and iHeartRadio.
- Amazon test report: Identifies and documents issues and recommendations for your device. Expect to receive the report between four to eleven weeks from the start of Amazon testing.
Use the AVS developer console to track where your product in the process, including the pass/fail status of each test. For status updates, watch for email notifications from Amazon.
Launch and post-certification Over-the-Air (OTA) updates
After your product passes all Amazon tests, it becomes eligible for certification and approval for launch. Test any post-launch changes to the firmware of your device before deploying any updates to products in the field through Over-the-Air (OTA) updates. If you make significant device updates, Amazon might require that you re-submit your device for recertification, incremental certification, or more self-testing.
Amazon requires recertification or incremental certification under the following conditions:
- Increasing in the memory footprint of the device.
- Altering the voice processing algorithms, such as the audio front-end (AFE) or wake word engine (WWE). Some examples include the following update types:
- Migrating from one WWE to another.
- Changing the AFE or WWE size.
- Migrating from one audio player to another.
- Changing the AFE details, echo canceling, or tuning.
- Implementing uncertified new Alexa features and capabilities.
- Coupling changes in a major release with Alexa interaction.
- Releasing updates for uncertified regions or languages.
If Amazon requests recertification, submit a set of release notes describing your changes to determine the necessary scope of testing by Amazon. Re-submit your completed Functional Self-test Checklist and Acoustic Self-test Checklist for all supported regions where you plan to deploy the OTA update.
If your product is implementing a major software update, such as an operating system upgrade or integrating Alexa Communications, you must submit a security reassessment report. Work with one of the Authorized Security Labs to retest your product against current security requirements and then provide a reassessment report. Ask your Amazon point of contact or email email@example.com for more details about the security reassessment process. For a list of test lab options, see Authorized Security Labs.
The following documents provide design, functional, and implementation guidance for AVS: