It’s no secret that customer satisfaction is key to Amazon’s success. It’s something we obsess about, and we constantly tweak everything from product selection to payment options to website design. We experiment, we measure, and we decide whether or not a change improves the user experience. One of the most powerful tools at our disposal is something we call a Web Lab, which allows us to test two (or more) different versions of a webpage, or roll out a new feature to a limited audience.

Perhaps this concept is less familiar to mobile application and game developers than to those who write web code, but it shouldn’t be. In fact, it’s so useful and important that Amazon’s A/B Testing service was one of the first we made available to mobile developers (If you haven’t tried the service yet, check out Getting Started with A/B Testing for a quick introduction).

Now a significant enhancement has just been released: Segmentation offers greater flexibility and control over who sees your experiments, allowing you to limit your tests to a set of users based on any dimensions you care to associate with them.

User dimensions

A “dimension” in this sense is simply a named characteristic whose value is a number or a bit of text. The UserProfile singleton object manages the dimensions you define, keeping track of their values for the current user. It’s these values that determine whether the current user is a member of a particular segment, and it’s entirely up to you how they are chosen, named, and initialized.

For example, say I want to test a new wine feature in my restaurant review app. I don’t want everyone to see it—just my beta testers—so I define a dimension called “isBetaTester” and initialize it based on some information I know about the current user:

        // Determine if the current user is actually participating in the beta,

        // based on my own definition of “participating.”

        boolean isBetaTester = validateUserAsBetaTester();

        // Set the dimension for the current user.

        UserProfile.addDimensionAsString("isBetaTester", String.valueOf(isBetaTester));

        // Initialize the Insights SDK to enable A/B Testing.

        AmazonInsights.withApplicationKey(MY_APPLICATION_KEY)

                      .withPrivateKey(MY_PRIVATE_KEY)

                      .withContext(getApplicationContext())

                      .initialize();

                 . . .

In this case, validateUserAsBetaTester() is a helper method I define to assist me in deciding whether the current user is a beta tester or not. This could be a preferences setting, or perhaps a check against an email whitelist.  Regardless, I’m responsible for setting the value, which I do before initializing the A/B Testing system.

Create a segment

Now that I’ve established a dimension to differentiate users, I need to define a segment that actually refers to it. The segment may be associated with any A/B test, limiting access to users who match (or don’t match) along the relevant dimension. From the Amazon Mobile App Distribution Portal, I navigate to the A/B Testing tab of my app, then click Add a New Segment.

Next, I fill in the segment information, paying special attention to the dimension information, since that must correspond to the name (“isBetaTester”) and type (String) I use in my application. In this case, I define a segment called “Beta Testers” that will include anyone whose isBetaTester dimension is a string equal to “true”:

Limit a test

With a segment defined, I can now use it to limit the scope of an A/B test. Let’s say I want to experiment with the length of user-submitted wine reviews. (I’d like to give people more space, but have to balance the storage required against the benefit of more detailed content.) When I create the test, the last setting allows me to specify the user segment I want to target. In this case, I choose my beta testers, whom I just identified:

When the test is started, beta users will begin to see variations of wineReviewLen, subject to the distribution I specify.  In my app, I can initialize a variable with the maximum allowable review length for the current user:

        ABTest.getVariationsByProjectNames("Wine Reviews")

            .withVariationListener("Wine Reviews", new VariationListener() {

                public void onVariationAvailable(Variation variation) {

                    maxReviewLen = variation.getVariableAsInt("wineReviewLen", 256);

                }

            });

For users outside the Beta Testers segment, the control variation will always be allocated. However, these users will never be included when the results of the experiment are calculated.

Complex segment criteria

In this simple example, a single boolean value was enough to distinguish segment members, but segmentation supports much more complicated criteria. You can easily attach more than one variable or change how variables are evaluated. String equality works as demonstrated (numeric equality is identical), but it’s also possible to create a segment based on a numeric range or set membership.

For example, perhaps I want to further segment my beta testers by those who have made a past purchase. I can create a numeric value called purchasesToDate and define the valid range as any non-zero value—note that I just have to start at 0 (exclusive) and omit the upper bound:

If I wanted to create a segment of customers in the Pacific Northwest, but only those who have used the app in 2012 or 2013, I could combine two tests for set membership:

Important note: users retain the variation they are initially allocated, even if their dimensions change later on.  This means that once a user has been identified as a member of one segment, they never move to another for the duration of the test, even if their dimensions change. This ensures a consistent experience; significant changes to features or interface could be jarring. (If a new test is started, the dimensions will be re-evaluated.)

That’s it!

Setting up segments is simple and adds a lot of flexibility to your experiments.  Give it a try the next time you’re considering A/B testing.  For information, be sure to check out the Insights documentation.