Skills Beta Testing for Alexa Skills

The skills beta testing tool can be used to test your Alexa skill in beta before releasing it to production. You can also use the skills beta testing tool to test changes to an existing skill, while still keeping the currently live version of the skill available for the general public. You can invite friends or family, your social network contacts, or other people for whom you have an email address to test your skill and provide feedback. You can send up to 500 invitations per skill. At any time, you can add testers, remove testers, or end a test. You can see tester feedback in the Amazon Developer Portal. You can get feedback on your skill throughout the testing cycle, and reduce costs by making fixes before you release the skill to production. Using skills beta testing can help increase your chances of skill success.

Security

In order to use the beta skill, the Alexa device used by the beta tester must be associated with the email address in the tester’s invitation.

If the beta tester does not have an Alexa device registered to that email address, then the tester can create an Amazon account using that email address and register it to their Alexa device. Alternatively, the tester can contact the developer and request that a separate invite be sent to the email address to which their device is registered. You must enter a contact email address, as described in Create a Beta Test for Your Skill, in order for testers to contact you.

Some beta testers may have opted out of receiving Amazon emails. If that is the case, these beta testers will not receive your invitation email. You can resolve this situation in two ways:

Set up a Beta Test for a Skill

  1. Log into the Amazon Developer Portal.

  2. Click Alexa at the top.

  3. Click Get Started > under Alexa Skills Kit.

  • Your list of Alexa skills is displayed.
  1. Click the skill you want to test.
  • The skill detail page is displayed.

If you have not filled out all six tabs on the left, your skill will show as Not yet eligible for skills beta testing.

Not yet eligible for skills beta testing
Not yet eligible for skills beta testing

Make a New Skill Eligible for Beta Testing

Your skill must meet specific criteria in order to be eligible for testing with the skills beta testing tool. These criteria include:

  • On the Skill Information tab, you must specify a Language, Name, and Invocation Name.

  • On the Interaction Model tab, you must define a language model.

  • On the Configuration tab, you must provide an endpoint for your skill’s service.

  • On the Publishing Information tab, you must provide all of the required information, and your Example Phrases must be successfully validated.

  • On the Privacy & Compliance tab, you must answer the questions as shown. If applicable, provide links to a Privacy Policy URL and a Terms of Use URL.

When all of this information is specified, your skill is eligible for testing with the skills beta testing tool. These steps also make your skill ready for certification.

Eligible for skills beta testing
Eligible for skills beta testing

Use the Skills Beta Testing Tool for a Beta Version of a Live Skill

When you have a skill that is live, you can still create a beta version of this skill. The testers who have access to the beta version will not have access to the live version. A tester may request removal from a beta test, and then after removal, this tester can re-enable the live skill in the Alexa app.

As described in Continuing to Work on Your Skill after Publication, once your skill is published, that version is what is available to the customer.

The live version and the development version of your skill are shown on the developer portal, as shown. Click Edit to change the development version. In this development version, use a different endpoint or Lambda function to ensure your original skill is not modified. Ensure all the criteria described in Make a New Skill Eligible for Testing are followed, and follow the steps to Create a Beta Test for Your Skill.

Live and Development Versions of a Skill
Live and Development Versions of a Skill

Create a Beta Test for Your Skill

  1. Click Beta Test Your Skill in the skills beta testing status box on the skill detail page. A testing page is displayed.
  2. Add tester email addresses to the Tester emails box. Remember that this email address must be the one that the tester uses for their Alexa account.
Add emails to box
Add emails to box
  • Separate each email address from the next with a semicolon or carriage return.
  • To upload a file of email addresses, click Bulk Upload. These addresses must be separated by semicolons or carriage returns, or both.
  1. Click Add testers.
  • The email addresses are validated, checked for duplicates, and added to the test. If one or more email addresses are not valid, an error is displayed. Correct the email addresses and click Add testers again.
  1. Review the list of eligible testers. You will also have the opportunity to add or remove testers after the test has started.

  2. In the Contact email field, you can enter a contact email address. You are encouraged to provide a way for your testers to give you feedback.

  3. Click Start Test.

Each tester will receive an email with a customized link inviting them to be beta testers for your skill. Amazon controls the messaging for this invitation.

The invitation email takes the following form, with the subject line of “You’re invited to test a new Alexa skill”.

Email Invitation to a Beta Test
Email Invitation to a Beta Test

When a beta tester clicks the link, they are prompted to log into their Amazon account and accept the Skills Beta Testing Terms and Conditions before they can start testing your skill.

Manage a Beta Test For Your Skill

Skills beta testing provides a dashboard, while a test is active, that shows which testers have accepted your invitation, and which allows you to add and remove testers. The test status is listed on the skill detail page, so you can tell at a glance if your beta test is currently active.

  • To open this dashboard, click Manage test skill in the skills beta testing status box on the skill detail page.
Manage a Beta Test For Your Skill
Manage a Beta Test For Your Skill

The Manage testers box shows the current invitation status for each tester. From here, you can add testers, remove testers, send reminders to testers, send feedback requests to testers, or end a test.

Remove a Tester

You can remove a tester from an ongoing test at any time.

  1. Open the skills beta testingpage for the skill.

  2. Click Remove tester for the tester you want to remove.

  3. Click Yes on the confirmation message.

The list of testers is updated.

If you remove a tester, that tester will receive an email that the test has ended, although the test does continue for all of the remaining testers. If the skill has both a live version and a development version, then a tester who has been removed will resume having access to the live version.

Send a Reminder to Testers

You can send a reminder to a tester for an ongoing test at any time. However, you are only allowed to send one reminder to each tester during the course of the beta test.

  1. Open the skills beta testing page for the skill.

  2. Click Send a reminder for the tester.

  3. Click Send reminder on the confirmation message.

The tester is sent an email, again with the messaging controlled by Amazon. This email contains the same content as the original invitation email, but the subject line is “[Reminder] You’re invited to test a new Alexa skill”.

Request Feedback From Testers

On the Manage testers page, if at least one tester has enabled a skill, a Request Feedback button is activated. Click to send testers with an enabled beta version a message asking for feedback. Ensure that your *Contact email address* has been set up to receive messages. The email that the testers will receive looks like this:

Email Requesting Feedback From Beta Testers
Email Requesting Feedback From Beta Testers

Add One or More Testers to an Existing Test

  1. Click Add Testers.

  2. Add the email addresses of the testers to the Tester emails box. You can enter the email addresses directly in the box, or click Bulk Upload to upload a CSV file with email addresses.

  3. Click Update testers to validate the email addresses. If necessary, correct the errors and click Update testers again.

The newly added testers are listed in the Manage testers box, along with the existing testers. They will be sent invitation emails the same as the original testers.

End a Test

You can choose to end a test at any time.

  1. In the developer portal, open the skill detail page for the skill.

  2. Click Manage test skills on the skill detail page.

  3. Click End test.

  4. Click Yes to the confirmation message.

Your testers will receive an email that the test has ended.

Start a New Test

You can start a new beta test for a skill after ending a previous test.

  1. In the developer portal, open the skill detail page for the skill.

  2. Click Start new test.

Tester information from the previous test is retained, and is used to prepopulate the Tester emails box. Double-check the listed testers to ensure that you want to keep them as testers, and add any new testers as desired. Start and manage the test as you would a new test.

FAQ for the Skills Beta Testing Tool

  1. Is the skills beta testing tool free?

Yes, the skills beta testing tool is a free service.

  1. How can I share my beta skill with my friends and family?

Obtain the email addresses that your friends and family use for their Alexa account. Add these email addresses to the beta test that you set up using the skills beta testing tool. The skills beta testing tool will send them the invitation emails by sharing a secure link to your skill generated by the skills beta testing tool. When a tester clicks the link, they will need to login with their Amazon credentials and opt in to test your skill. When the tester does opt in, the skill is automatically enabled on any device that’s associated with that Amazon account.

  1. How long can the beta version be live?

The beta version is live for a maximum of 90 days, at which point the test ends automatically and all participants are notified by email that the test has ended. The skill developer can also choose to end the beta at any time during this 90-day period. Note that the skill developer can create a new test if desired, which will require the same steps as creating the original test.

  1. What is the maximum number of beta testers I can invite per skill?

You may invite up to 500 testers.

  1. Can I have a beta test version of a skill at the same time that I have a live version of a skill?

Yes, both a live version, which is available to the general public, and a beta test version, available only to selected testers, can co-exist. However, beta testers can only access the beta test version once they have enabled it, unless you remove the tester from the test, or a tester chooses to remove themself from the test, or the beta test is ended.

  1. Can I prompt testers for feedback?

If at least one tester has enabled the skill, then on the Manage skill page, you can click ‘Request Feedback’ button. Each tester who has enabled the skill will get an email request to send feedback to your contact email address.

  1. Can I customize the email messages sent to testers?

You cannot customize these messages which are sent by Amazon. However, you can communicate with your testers outside the scope of the skills beta testing tool.

  1. Can I publish different beta versions of the skill to multiple groups?

No, you cannot publish multiple version of the skill.

  1. Are the emails optimized for mobile?

Yes, the email templates are optimized for mobile devices.

  1. How can the beta testers provide feedback?

The testers will be able to provide feedback using the Give Feedback email link available within the invitation email. In addition, if the skill developer clicks the Request Feedback button on the Manage testers page, then another email requesting feedback is sent to those testers who have enabled the original beta skill link.

  1. How can beta users differentiate between a live skill and a beta skill?

The beta skill will be tagged with a “dev” tag on the skill icon.

  1. Does the skill need to go through certification to be able to launch it as a beta?

The skill does not require any certification to start a beta test.

  1. Can I update the beta version of the skill once I have published my beta?

Yes, you can update the beta version of the skill, after you have published the beta.

  1. Can the beta testers unsubscribe to the invitation emails?

Yes, beta testers will be able to unsubscribe from emails from the skills beta testing tool.

  1. Will developers be able to use actual Account linking with the skills beta testing tool?

Yes, developers will be able to use the account linking feature with the beta testing tool, just as for live skills.

  1. How do I remove a tester from a beta test?

To remove a tester, click the Remove Tester button next to their email address. You can also choose to end the test, which effectively removes access for all testers.

Next Steps