Product Testing Overview
Amazon Alexa users expect a high quality and consistent experience across all products. To help you meet this bar and build the best possible Alexa integration, this document explains the product testing process, and provides an overview of the typical Alexa Voice Service (AVS) development lifecycle.
- Product Testing
- Step 4: Post-certification updates (OTA)
Product testing is broken into two steps: self-test, and product submission. During self-test, you will use a series of checklists provided by Amazon to ensure that your product meets functional and user experience (UX) requirements. If your product supports music and/or is far-field enabled, additional testing is required prior to submitting your product to Amazon for evaluation.
After you've completed the self-test checklists, they must be submitted to Amazon for evaluation prior to commercial distribution of your AVS product. Please work with your Amazon point of contact to schedule product submission and testing. This process takes approximately 10 weeks.
Step 1: Complete Self-Test Checklists
For access to these self-tests and the certification intake form, please contact your Amazon POC or follow the instructions for how to launch with AVS.
Step 2: Submit Your Product for Testing
After you've completed the self-test checklists, please work with your Amazon point of contact to submit the results and schedule testing. After review, authorization to ship products along with shipping instructions will be provided to you directly from the testing team.
The table below lists the number of devices to be submitted to Amazon for testing, based what features your product supports:
|Test||# of Products||Notes|
|Music Certification||1||If supported|
|Far-Field/Hands-Free evaluation||2||If supported|
Step 3: Testing by Amazon
This is a high-level overview of the process that begins once Amazon receives your product for evaluation:
- Your product will be verified using an Acceptance Test to ensure that it meets basic functional requirements. This tests a subset of the items in the Functional Checklist completed during self-testing.
- Your product will be tested to ensure that it meets the requirements specified in the Amazon Developer Services Agreement, the Functional Requirements for AVS and User Experience Guidelines.
- If access to third party music or radio services using the Alexa Service is supported by your product, then music certification will be scheduled. This is required by Music Service Providers (MSP), including Amazon Music, TuneIn and iHeartRadio. Note: As new MSPs are on-boarded, music certification will be expanded to include any additional requirements.
- If your product is Far-Field/Hand-free enabled and uses a wake word to initiate interactions with Alexa, testing will be scheduled and performed at the same time as all other testing.
- Identified issues and recommendations will be documented and returned to you within 4-11 weeks from the start of testing.
NOTE: Your Amazon point of contact will provide you with a detailed timeline outlining the testing process prior to product submission.
If your product meets all testing requirements, it will be approved for launch. You will be contacted within 48-hours by your Amazon PoC when your product passes testing. At this time, any additional assets required to launch will be provided.
Step 4: Post-certification updates (OTA)
Any updates or changes to the firmware must be tested by the product owner before being deployed to products in the field (OTA updates). If significant updates are made, devices that were previously certified by Amazon may be required to be re-submitted for certification, incremental certification, or self-testing.
Re-certification or incremental certification by Amazon is required under the following conditions:
- Changes cause an increase in the memory footprint of the device.
- Changes are made to voice processing algorithms such as the audio front end (AFE) or wake word engine (WWE). Examples include:
- Moving from one WWE to another.
- Moving from one AFE or WWE size to another.
- Moving from one audio player to another.
- Making changes to the AFE affects gain, echo canceling, or tuning.
- Adding new Alexa features and capabilities which were not previously certified.
- Including changes in a major release that is tightly coupled with Alexa interaction.
- Releasing updates in supported regions or languages which were not previously certified.
When re-certification is required, release notes describing the changes must be provided to determine the scope of testing by Amazon. Re-submit your successfully completed Functional Self-Test Checklist and Acoustic Self-Test Checklist for all supported regions where the OTA update will be released.
The documents listed below provide design, functional, and implementation guidance:
- Audio Hardware Configurations
- Functional Requirements for AVS
- UX Design Overview
- Marketing and Brand Guidelines
- API Overview