Our A/B Testing service just added support for A/B/n tests, allowing more than two variations in a single experiment. This feature makes it even easier to quickly optimize your app in the field—without having to republish your APK. A/B/n testing saves you time since one experiment can now test up to five variations at once.

For example, suppose you have four ideas for how to label a Buy button – Buy, Purchase, Add to Cart, and Get it Now! – and want to know which label leads to the most purchases. You start by setting up an A/B test as you normally would (if you’re new to A/B testing on mobile, learn more about getting started) and click Add Variation for each new variation:

A/B/n testing saves time, since standard A/B testing of four variations would require completing multiple A/B tests one after the other. The experiment above, for example, would require three separate tests, with the winning test becoming the new control each time.

A/B/n testing lets you compare all four variations at once, so only one test is necessary:

When planning an A/B test it is important to consider the size of the daily active user base you want to affect and how long you can wait for the test to complete. If a variation’s slice of the data is small, the A/B test may take a long time to achieve statistical significance. Apps with a large audience can likely distribute most customers to Variation A (Control) without sacrificing test time or significance. Apps with a smaller audience will want to distribute tests evenly across all variations as shown below:

After you identify the winning variation, the A/B Testing service allows you to launch it to customers with a simple button click on the Mobile App Distribution Portal. There’s no need to republish your app. You can simply indicate which variation the service should distribute to customers from now on.

If you’re an Android developer, explore this update to the free Amazon A/B Testing service. We’re interested in hearing your feedback on A/B Testing. Please take this short survey to help us improve the service.