When you submit your app to the Amazon Appstore, the app must pass tests before it can be published. During development, you can use the following test criteria as a guide for your own quality assurance testing.
At a high level, your app should have the following characteristics:
- Have a simple and well-thought-out user experience that does not cause user confusion.
- Be thoroughly tested and free of bugs, crashes, and obvious defects.
- Fully utilize the screen area of the Fire tablet, taking into consideration both the notification bar and the dynamic soft key menu behavior.
- Provide visual indication that the user's action has succeeded or failed.
- Fail gracefully if the user tries to access functionality that is not available on the Fire tablet, for example by displaying an error message such as This feature is not currently available.
- Be free of visual defects, including missing, distorted, or pixelated images; misaligned user interface elements; or illegible text.
- Have grammatically correct user interface text with correct spelling.
Group 1: Capability Testing
This group of tests is designed to make sure that your app does not use device capabilities that are not present on Fire tablets. It also ensures that, if your app does request device permissions that are not present, it handles the error condition in a user-friendly manner.
For best compatibility, your app should not use any of the below permissions. If your app includes the listed permission statement, you must ensure it degrades gracefully and handles the error condition in an elegant and user-friendly manner. That is, lacking the permission, your app must not force close or crash, and it should provide a friendly error message.
Group 2: App Behavior on Device
Your app must interact correctly with the Fire tablet and its operating environment. To complete this group of tests, install the app on a Fire tablet and launch.2.1 Launch Without Error
- Test: Successfully launch the app from the app library.
- Expected Results: The app launches on the Fire tablet and will advance the user into the app's interface without any errors.
- Test: Observe that loading screens taking longer than 15 seconds display a "loading" message.
- Expected Results: If loading takes longer than 15 seconds, the app must indicate this to the user (through loading screen, progress bar, etc).
- Test: If the app uses fullscreen, observe that the soft keys toolbar is minimized (with the fullscreen handle visible) and that the Settings toolbar is hidden.
- Expected Results: When the app uses fullscreen, the soft keys toolbar should be minimized (with the fullscreen handle visible) and the Settings toolbar should be hidden.
- Test: If the app does not use fullscreen, observe that the soft keys toolbar and the Settings toolbar are present.
- Expected Results: When the app does not use fullscreen, the soft keys toolbar and the Settings toolbar should be present.
- Test: Observe the soft key menu button expands for more soft keys toolbar options.
- Expected Results: Use of the Menu button is not required. But when in use, tapping the Menu button should not cause the app to force close.
- Test: Observe the behavior of the Back button when pressed within the app, reverting back to a previous screen.
- Expected Results: Use of the Back button is not required. But when in use, the Back button should not cause the app to force close.
- Test: Observe the behavior of the Home button when pressed within the app.
- Expected Results: Use of the Home button is required and should take the user back to the Home screen of the Fire tablet.
- Test: Observe the behavior of the Search button when pressed within the app.
- Expected Results: Use of the Search button is not required. But when in use, the Search button should not cause the app to force close and allow the user to search within the app.
- Test: Observe the behavior of the Settings toolbar when invoked, and observe the app's performance when changing the settings (for example, volume).
- Expected Results: Users should have the ability to change Fire tablet settings while running the app. Use of the Settings menu should not cause the user to lose state, cause a force close, or impede the user's experience in any way.
- Test: Verify that the soft keys toolbar does not interfere with the app or cover important UI elements.
- Expected Results: The soft keys toolbar should not obstruct any elements of the app, for example, buttons, text, or important graphics.
- Test: Search for areas where users can become trapped, that is, unable to navigate using either onscreen elements or the soft keys buttons.
- Expected Results: Users should be able to navigate the interface of the app without becoming locked into any one screen. In addition, users should not be forced to exit the app using the Home soft key.
- Test: Invoke hibernation mode of the device by pressing the Power button.
- Expected Results: When the device is in hibernation mode, the app should return to its previous state when it comes out of hibernation mode, saving all user data and progress. To save their state, most apps should pause; some apps, however, such as alarms, radios, or music players, should keep running when the device is hibernating. Also, the app may pass if the loss of user state is insignificant.
- Test: Observe the behavior of the app when the network connection has been lost while a download is in progress.
- Expected Results: Wake lock should be active for large downloads (large downloads >= 5 minutes). If Wake lock has not been implemented and the device has gone into hibernation mode, when the user returns to the app they must be informed via a message that the download has failed.
- Test: Ensure that the overall performance of the app sustains a usable and pleasant user experience.
- Expected Results: Apps should sustain a high frame rate without dropping below 25 fps for an extended period of time. Apps should not impede the user's ability to interact with an app. For example, users should not experience a noticeable delay when typing in an app. The app should not force close, crash, or hard lock while in use. The recommended average frame rate is 55-60 fps, especially for apps that involve fast-moving graphic objects.
- Test: Observe any reproducible or non-reproducible force closes.
- Expected Results: The user must be able to interact with the app's UI and features without any force closes.
- Test: Observe any reproducible or non-reproducible hard locks.
- Expected Results: The user must be able to interact with the app's UI and features without any hard locks.
- Test: Repeatedly press buttons or other graphical elements on the screen.
- Expected Results: The app should not force close when users rapidly press a button.
- Test: Observe the graphical UI elements within the app and verify that no distorted, pixelated, misalignments, or other graphical anomalies occur.
- Expected Results: The app's graphics should not appear distorted, pixelated, misaligned, or contain any other graphical anomalies. The user experience should not suffer because of poor-quality graphics.
- Test: Observe key presses for UI displacement.
- Expected Results: Verify that the location of UI graphics correspond accurately to where their interactive areas exist on the touchscreen. For example, in some cases, users may have to press 20px above where a button is displayed—this would be a fail.
- Test: Observe the behavior of external links within the app.
- Expected Results: External links within the app should not break any functionality of the app. After returning to the app from an external link, the app should have saved the user state and progress. Links may appear to fail because they direct users to apps that are not yet live. This behavior is expected, but the links should work once the apps are live.
- Test: Observe the amount of screen that the app occupies.
- Expected Results: Apps should occupy 100% of the screen to be fully compatible. But apps may still pass as long as they fill 80% of the screen and are centered horizontally and vertically on the screen; apps should not be located in one corner of the screen. Also, the unfilled portions of the screen must be free from graphical anomalies.
- Test: Observe the behavior of in-app volume and mute controls.
- Expected Results: Any volume and mute functions built into an app should function as expected. If muted, then the sound should no longer play.
- Test: Observe the placement and text within user input data fields.
- Expected Results: Input boxes should fully display the text that the user has typed without being cropped or unreadable.
- Test: Observe the behavior of the app when the device is rotated between landscape and portrait modes. Repeat as needed throughout the app.
- Expected Results: When the screen orientation is rotated, the graphics onscreen should adjust properly to fill the new layout. Text, graphics, and buttons should appear without error while being fully functional from one layout to another. Note that not all apps allow the screen to be rotated. Also note that pictures and videos can be the exception to the rule due to their fixed aspect ratios.
- Test: Observe the functionality of any social networking login pages.
- Expected Results: Users should be able to interact with social networking login pages, allowing users to log into their account. Ensure that the login and approval functions work and display as intended.
- Test: Observe the behavior of buttons within the app that submit or attempt to share user data.
- Expected Results: Apps that use SMS or MMS to share user data should not force close. These apps should be able to gracefully fail with a relevant error message.
- Test: Observe the social networking functions and their interactions with the app.
- Expected Results: If the app features any social networking integration (Facebook, Twitter, etc.), verify that content is able to be uploaded/shared across these mediums as expected. Account Creation: If an external service provided by the app requires the user to create an account, verify that an account may be created and used with the app.
- Test: Observe external links that link to the Amazon Appstore for Android.
- Expected Results: All links that point to other apps should link correctly to the Amazon Appstore. If a link points directly to an app on the Amazon Appstore and is not currently live on the store, then it is acceptable for that link to fail.
- Test: Uninstall the app from the device.
- Expected Results: The app should be uninstalled from the device.
- Ensure a smooth and quick exit occurs when the Home button is pressed.
- Expected Results: The app should not take more than 2 seconds to exit when the Home button is pressed.
- Test: Ensure quick installation of app resources.
- Expected Results: Ideally, the app will not take more than 15 seconds to install. Make sure the unused resources are removed from the app's binary. This will reduce the binary size and help in quick installation.
- Test: Ensure the app uninstalls quickly and removes all artifacts.
- Expected Results: Ideally, the app will take no more than 2 seconds to uninstall. The app should create minimal artifacts on the disk. This will help ensure faster clean-up while uninstalling the app.
- Test: Ensure that the app's orientation is correct following device orientation setting.
- Expected Results: If device orientation is locked in device settings, app orientation should not change upon rotating the device. The app should open in its default orientation or as per the locked orientation in the device. The orientation should not change automatically on rotating the device if it is locked in the device.
- Test: Ensure that the app is not consuming battery excessively.
- Expected Results: Battery-intensive operations should be optimized to ensure that power consumption remains low.
- Test: Ensure that the device temperature does not increase to the point of discomforting the user, even upon prolonged use.
- Expected Results: The app should not heat up the device such that it results in discomforting the user when it is held in both hands. If continuous usage of the app causes the device temperature to reach 40 degree Celsius from normal ambient temperature (33 degrees Celsius) during a period of 1 hour, then it would be considered a fail. Make sure the intensive operations are optimized to ensure that the temperature remains low.
- Test: All possible gestures should be handled as per their standard behavior. Expected Results: Gestures like pinch (in/out), swipe (left/right/up/down), and press and hold, should work as expected.
- Test: Ensure the game preserves its state after USB plug-in / plug-out.
- Expected Results: When the device is in USB mode, the app should return to its previous state when it comes out of USB mode, saving all user data and progress.
- Test: Ensure there are no exceptions thrown from the app.
- Expected Results: We recommend you analyze the device logs to see if the app is throwing any errors or exceptions. Ideally, the app should not throw any errors or exceptions in the background.
Group 3: Core App Functionality
In general, your app should "work the way it is designed." This is a general statement, but core functionality tends to be overlooked while being one of the most important aspects of testing. The following tests relate to such things as games keeping scores correctly, following game rules, and so forth.
3.1 Correct App for a Fire Tablet
- Test: Observe the core functionality of the app.
- Expected Results: Determine the main purpose of the app and whether the app can be used for this purpose on the Fire tablet (for example, a photo-editing app must be able to edit photos).
- Test: Observe any missing functions that were reported as a feature of the app.
- Expected Results: The features mentioned in an app's description should be available on Fire tablets.
- Test: Observe any features used in the app that are not found on Fire tablets.
- Expected Results: Any features of the app that are not available on Fire tablets should degrade gracefully with a message rather than a force close. It should also be noted that apps that do degrade gracefully should still provide value to the end user. If an app's core functions are the only value to the app, then the app should be considered a fail. For example, if a photo-taking app does not allow the user to take photos, but it messages the user that the camera is unavailable without it being a force close, the app still fails. If the main function of an app is still accessible to the user, when other functions are not, then it could be considered a pass. For example, if a bus route app messages the user that Google Maps is unavailable and the user still has the ability to view the bus routes and the bus time tables, it can be considered a pass.
- Test: Observe the featured use of an external e-mail app as it relates to the core functionality of the app when e-mail has and has not yet been set up on the device.
- Expected Results: If one of the core functionalities of the app is to send e-mails with an external e-mail client, then it should be tested without a user's e-mail previously set up through the external e-mail client. It has been observed that apps will force close when attempting to call an external e-mail client. The expected result is that the app should not force close. The app can disable the e-mail feature with or without a message notifying the user.
- Test: Observe the app's behavior when changing the settings of software-related options (for example, game difficulty, filtering, and display settings).
- Expected Results: Tested settings should function as anticipated, changing the related function within the app.
- Test: Observe the app's behavior when changing the settings of hardware-related options (for example, volume, accelerometer, and landscape vs. portrait settings).
- Expected Results: Tested settings should function as anticipated, changing the related function within the app.
- Test: Observe the app's ability to save user state or data after exiting the app.
- Expected Results: If the app has modes or features that save user data, this save data must be stored and remain accessible after exiting and restarting the app.
- Test: Ensure the layout of your app looks appropriate for a Fire tablet screen.
- Expected Results: Your app should not look like the mobile phone version of an app that has been scaled up for a Fire tablet screen. The layout and UI elements such as buttons, text input box, fonts, images, and so forth, should be optimized for a Fire tablet. Also, touch targets should not be too small. The objects should be identified easily. The size should be large and spaced enough for appropriate thick finger touch.
Note: As per Android UI development guidelines, the touch area should be at least 7*7 (mm).3.9 Accessing External Applications
- Test: Ensure app dependencies are handled correctly. Expected Results: If your app relies on another app to perform certain tasks, such as sending a file, and there are no apps on the device to perform this action, the main app should not force close.
- Test: Ensure smooth 2D graphics rendering throughout the app.
- Expected Results: Hardware acceleration should be enabled in the manifest file.
To enable the hardware-accelerated 2D graphics, you should edit the AndroidManifest.xml file and add the following attribute to the <application /> tag:
- Test: Install the old version of the app and save some user data. Install latest version of the app over the old version. Observe whether the data saved in the old version of the app is accessible in new version.
- Expected result: The app should preserve user data, such as game data, content purchased through In-App Purchasing, saved settings, downloaded content, login credentials, searches, saved chats, bookmarks, saved pages, and so on.