App Performance Scripts

Performance testing is a process for testing app performance metrics such as compatibility, reliability speed, response time, stability, and resource usage in Fire tablets and Fire TV devices. This testing identifies the performance bottlenecks and improves the performance by communicating with developers. Testing for mobile applications is heavily Key Performance Indicator (KPI) driven wherein a specific set of KPIs and steps are executed on Amazon devices and metrics are calculated using Amazon/generic device resources such as logs.

This page captures the steps and snippets required to test performance KPIs for apps submitted to Amazon Appstore. The base KPIs targeted for testing are latency, ready-to-use and memory, or video streaming.

Setup
Test strategy
Cool launch until first frame
- Cool iterations
- Device cool vital logs
Warm launch until first frame
- Warm iterations
- Device warm vital logs
Ready-to-use
Memory KPI
Additional Links
- Writing a KPI log file

Setup

Before you start, install the following software packages on your development computer:

In addition to the installing the software packages, you will need to:

Set up a path for JAVA_HOME and ANDROID_HOME folders.
Enable Developer Mode on the device and enable USB Debugging, for instructions see Enable Developer Mode.
Capture the device udid or Device_Identifier, for instructions see DSN.

Test strategy

During the testing, the app will be launched and force stopped several times using app launcher intent or The Monkey tool. Certain necessary actions should be performed between each iteration such as capturing ADB logcat, performing navigation actions (ready-to-sse and memory KPIs), capturing timer values from vitals/ADB logs, and capturing memory and RAM usage before force stopping the app. This loop will continue for the number of iterations configured. As the test results can be impacted by network conditions, system load, and other factors, multiple iterations should be used to average out the interference from external factors. For latency KPIs, we recommend running a minimum of 50 iterations, for ready-to-use KPIs we recommend a minimum of 10 iterations, and for memory KPIs we recommend a minimum of 5 iterations

The logs, screenshots, and other artifacts captured from the performance tests can be used later for debugging or other critical information. The Appium device object is force stopped as part of tear-down.

Get device type

For an example of test methods that can be added to a test automation script, use the following code sample.

Java
Kotlin

Java

public String get_device_type() {
  String deviceType = null;
  try (BufferedReader read = new BufferedReader(new InputStreamReader
                    (Runtime.getRuntime().exec
                    ("adb -s "+ DSN +" shell getprop ro.build.configuration")
                    .getInputStream()))) 
  {
            String outputLines = read.readLine();
            switch (outputLines) {
                case "tv":
                    deviceType = "FTV";
                    break;
                case "tablet":
                    deviceType = "Tablet";
                    break;
  }
  catch (Exception e) {
     System.***out***.println("Exception while getting device type info: " + e); 
  }
  return deviceType;
}

Kotlin

import java.io.BufferedReader
import java.io.InputStreamReader

fun getDeviceType(): String? {
    var deviceType: String? = null
    try {
        BufferedReader(InputStreamReader(Runtime.getRuntime().exec("adb -s $DSN shell getprop ro.build.configuration").inputStream)).use { read ->
            val outputLines = read.readLine()
            when (outputLines) {
                "tv" -> deviceType = "FTV"
                "tablet" -> deviceType = "Tablet"
            }
        }
    } catch (e: Exception) {
        println("Exception while getting device type info: $e")
    }
    return deviceType
}

Generating main launcher activity of the app

Use the following code sample to generate the main launcher activity. This method fetches the main activity of the app being tested and generates the main launcher activity by combining the app package and main activity.

Java
Kotlin

Java

try (BufferedReader read = new BufferedReader(new InputStreamReader
                    (Runtime.getRuntime().exec("adb -s "+ DSN +" shell pm dump"+ appPackage +" | grep -A 1 MAIN").getInputStream()))) {
            String outputLine = null;
            while ((line = read.readLine()) != null) {
                if (line.contains(appPackage + "/")) {
                    break;
                }
            }
            
            outputLine = outputLine.split("/")[1];
            outputLine = outputLine.split(" ")[0];
            String appLauncherIntent = appPackage + "/" + outputLine;
            return appLauncherIntent
}
catch (Exception e) {
        System.out.println("There was an exception while generating App Main Activity" + e);
}            

Kotlin

import java.io.BufferedReader
import java.io.InputStreamReader

try {
    val process = Runtime.getRuntime().exec("adb -s $DSN shell pm dump $appPackage | grep -A 1 MAIN")
    val inputStream = process.inputStream
    val reader = BufferedReader(InputStreamReader(inputStream))
    var line: String? = null
    var outputLine: String? = null
    while (reader.readLine().also { line = it } != null) {
        if (line!!.contains("$appPackage/")) {
            break
        }
    }
    outputLine = outputLine!!.split("/")[1]
    outputLine = outputLine!!.split(" ")[0]
    val appLauncherIntent = "$appPackage/$outputLine"
    appLauncherIntent
} catch (e: Exception) {
    println("There was an exception while generating App Main Activity $e")
}

Launching an app with the main launcher activity (app intent)

Use the following code sample to launch an app using app intent or app package and main activity.

Java
Kotlin

Java

try (BufferedReader read = new BufferedReader(new InputStreamReader
                    (Runtime.getRuntime().exec("adb -s "+ DSN +" shell am start -n " + appActivity).getInputStream()))) {
            String deviceName = getDeviceName(DSN);
            String line;
            while ((line = read.readLine()) != null) {
                if (line.startsWith("Starting: Intent")) {
                    System.out.println("App Launch successful using App Intent - " + appIntent);
                    break;
                } else if (line.contains("Error")) {
                    System.out.println("App Launch Error");
                }
            }
        } catch (Exception e) {
            System.out.println("There was an exception while launching app using App Intent" + e);
        }

Kotlin

import java.io.BufferedReader
import java.io.InputStreamReader

val process = Runtime.getRuntime().exec("adb -s $DSN shell am start -n $appActivity")
val inputStream = process.inputStream
val reader = BufferedReader(InputStreamReader(inputStream))

try {
    val deviceName = getDeviceName(DSN)
    var line: String?
    while (reader.readLine().also { line = it } != null) {
        if (line!!.startsWith("Starting: Intent")) {
            println("App Launch successful using App Intent - $appIntent")
            break
        } else if (line!!.contains("Error")) {
            println("App Launch Error")
        }
    }
} catch (e: Exception) {
    println("There was an exception while launching app using App Intent $e")
}

Launching an app using the Monkey tool

Use the following code to launch an app using The Monkey tool.

Java
Kotlin

Java

try {
     String monkeyCommand = null;
     if (DEVICE_TYPE.equals(FTV)) {
          monkeyCommand = " shell monkey --pct-syskeys 0 -p "
     }
     else {
          monkeyCommand = " shell monkey -p "
     }
     
     BufferedReader launchRead = new BufferedReader(new InputStreamReader
            (Runtime.getRuntime().exec("adb -s "+ DSN + monkeyCommand + appPackage +" -c android.intent.category.LAUNCHER 1").getInputStream()));
      
     String line;
     while ((line = launchRead.readLine()) != null) {
         if (line.contains("Events injected")) {
             System.out.println("App Launch successful using Monkey Tool - " + appPackage);
             launchRead.close();
             return true;
         } 
         else if (line.contains("Error") || line.contains("No activities found")) {
             System.out.println("Error while launching app through Monkey Tool, using Intent to launch");
             launchRead.close();
             return false;
         }
     }
}
catch (Exception e) {
     System.out.println("There was an exception while launching the app using Monkey" + e);
     return false;
}  

Kotlin

try {
     val monkeyCommand: String?
     if (DEVICE_TYPE == FTV) {
          monkeyCommand = " shell monkey --pct-syskeys 0 -p "
     }
     else {
          monkeyCommand = " shell monkey -p "
     }
     val launchRead = BufferedReader(InputStreamReader(
            Runtime.getRuntime().exec("adb -s $DSN $monkeyCommand $appPackage -c android.intent.category.LAUNCHER 1").inputStream))
     var line: String?
     while (launchRead.readLine().also { line = it } != null) {
         if (line!!.contains("Events injected")) {
             println("App Launch successful using Monkey Tool - $appPackage")
             launchRead.close()
             return true
         } 
         else if (line!!.contains("Error") || line!!.contains("No activities found")) {
             println("Error while launching app through Monkey Tool, using Intent to launch")
             launchRead.close()
             return false
         }
     }
}
catch (e: Exception) {
     println("There was an exception while launching the app using Monkey $e")
     return false
}

Force stopping an app

Use the following code sample to force stop an app.

Java
Kotlin

Java

try {
       Runtime.getRuntime().exec("adb -s "+ DSN +" shell am force-stop " + appPackage);
       System.out.println("App force stopped - " + appPackage);
} 
catch (Exception e) {
       System.out.println("There was an exception in force stopping app" + e);
}

Kotlin

try {
    Runtime.getRuntime().exec("adb -s ${DSN} shell am force-stop ${appPackage}")
    println("App force stopped - $appPackage")
} catch (e: Exception) {
    println("There was an exception in force stopping the app: $e")
}

Latency KPI

Latency until first frame is the time taken by an app from launch to the first frame drawn by the app. This is used to measure the app performance while launching it in different ways. This KPI is used to replicate user behavior and improve the customer experience.

Before launching an app programmatically, you need some basic parameters such as app’s package name, main_activity, or other specifics.

Java

public String COOL_APP = "cool_app_launch_time";
public String COOL_ACTIVITY = "cool_activity_launch_time";
public String WARM_APP_WARM = "warm_app_warm_transition_launch_time";
public String WARM_APP_COOL = "warm_app_cool_transition_launch_time";
public String AM_ACTIVITY_LAUNCH_TIME = "am_activity_launch_time:";
public String WM_ACTIVITY_LAUNCH_TIME = "wm_activity_launch_time:";
public String WARM_ACTIVITY_LAUNCH_TIME = "performance:warm_activity_launch_time:";
public String AM_FULLY_DRAWN = "am_activity_fully_drawn_time:";
public String WM_FULLY_DRAWN = "wm_activity_fully_drawn_time:";

Kotlin

const val COOL_APP: String = "cool_app_launch_time"
const val COOL_ACTIVITY: String = "cool_activity_launch_time"
const val WARM_APP_WARM: String = "warm_app_warm_transition_launch_time"
const val WARM_APP_COOL: String = "warm_app_cool_transition_launch_time"
const val AM_ACTIVITY_LAUNCH_TIME: String = "am_activity_launch_time:"
const val WM_ACTIVITY_LAUNCH_TIME: String = "wm_activity_launch_time:"
const val WARM_ACTIVITY_LAUNCH_TIME: String = "performance:warm_activity_launch_time:"
const val AM_FULLY_DRAWN: String = "am_activity_fully_drawn_time:"
const val WM_FULLY_DRAWN: String = "wm_activity_fully_drawn_time:"

General ADB logcat commands used

Use the following code sample to capture the ADB logs with threadtime to get the app’s performance latency metrics.

Java
Kotlin

Java

public String ADB_LOGCAT_DUMP = "adb shell logcat -v threadtime -b all -d";
public String ADB_LOGCAT_CLEAR = "adb shell logcat -v threadtime -b all -c";


//Below method is used to append adb commands with the DSN value of the Device
public Process adb(String DSN, String message) {
 Process process = null;
  try {
      process = Runtime.getRuntime().exec("adb -s " + DSN + message);
    } 
  catch (Exception e) {
      System.out.println("Exception while executing adb commands" + e);
  }
  return process;
}

Kotlin

public const val ADB_LOGCAT_DUMP: String = "adb shell logcat -v threadtime -b all -d"
public const val ADB_LOGCAT_CLEAR: String = "adb shell logcat -v threadtime -b all -c"

//Below function is used to append adb commands with the DSN value of the Device

fun adb(DSN: String, message: String): Process? {
    return try {
        Runtime.getRuntime().exec("adb -s $DSN$message")
    }
    catch (e: Exception) {
        println("Exception while executing adb commands$e")
        null
    }
}

Capture timer values from vital buffer logs

Use the following code sample to filter out the performance latency timer value from the vital buffers.

Java
Kotlin

Java

public int get_vitals_timer(File logFile, String appPackage, String metricSearch) {
   BufferedReader reader = null;
   String line_Metric;
   int timer = 0;
   
   try {
       reader = new BufferedReader(new FileReader(logFile))
        while ((line_Metric = reader.readLine()) != null) {
                if (line_Metric.contains("performance:" + metricSearch) 
                && line_Metric.contains("key=" + appPackage)) {
                    timer = splitToTimer(line_Metric);
            }
    }
    catch (Exception e) {
        System.out.println(e);
    } 
    finally {
        reader.close();
    }
   return timer;     
} 

Kotlin

import java.io.File

fun getVitalsTimer(logFile: File, appPackage: String, metricSearch: String): Int {
    var reader: BufferedReader? = null
    var lineMetric: String
    var timer = 0
    try {
        reader = logFile.bufferedReader()
        while (reader.readLine().also { lineMetric = it } != null) {
            if (lineMetric.contains("performance:$metricSearch") && lineMetric.contains("key=$appPackage")) {
                timer = splitToTimer(lineMetric)
            }
        }
    } catch (e: Exception) {
        println(e)
    } finally {
        reader?.close()
    }
    return timer
}

Capture timer values from activity manager buffer logs

The following code will filter out the timer values of the app using the activity or window manager buffer.

Java
Kotlin

Java

public int get_displayed_timer(File adb_log_file, String appPackage) {
        BufferedReader br = null;
        int timer = 0;
        String displayedMarker = null;
        
        try {
            br = new BufferedReader(new FileReader(adb_log_file));

            while ((displayedMarker = br.readLine()) != null) {
                if ((displayedMarker.contains(AM_ACTIVITY_LAUNCH_TIME) || 
                    displayedMarker.contains(WM_ACTIVITY_LAUNCH_TIME) ||
                    displayedMarker.contains(WARM_ACTIVITY_LAUNCH_TIME)) && 
                    displayedMarker.contains(appPackage))
                        break;
            }

            if (displayedMarker != null) {
                displayedMarker = displayedMarker.split("]")[0].split("\\[")[1];
                ffValue = Integer.parseInt(displayedMarker);
            } 
        } catch (Exception e) {
            System.out.println(e);
        } finally {
            br.close();
        }
        return timer;
    }

Kotlin

fun getDisplayedTimer(adbLogFile: File, appPackage: String): Int {
    var br: BufferedReader? = null
    var timer: Int = 0
    var displayedMarker: String? = null

    try {
        br = BufferedReader(FileReader(adbLogFile))

        while (displayedMarker != null) {
            if (displayedMarker.contains(AM_ACTIVITY_LAUNCH_TIME) ||
                    displayedMarker.contains(WM_ACTIVITY_LAUNCH_TIME) ||
                    displayedMarker.contains(WARM_ACTIVITY_LAUNCH_TIME)) &&
                    displayedMarker.contains(appPackage)) {
                break
            } else {
                displayedMarker = br.readLine()
            }
        }

        if (displayedMarker != null) {
            displayedMarker = displayedMarker.split("]")[0].split("\\[")[1]
            timer = Integer.parseInt(displayedMarker)
        }
    } catch (e: Exception) {
        println(e)
    } finally {
        br?.close()
    }

    return timer
}

Cool launch until first frame

Latency cool launch until first frame is launching the app from the home screen or launching the app after force stopping and then measuring the time taken to draw the first frame of the app. This metric measures the performance of the app in this type of launch because the app typically takes more time to launch after a force stop to load services and keep them in the cache.

To measure the latency cool launch until first frame:

Run the logcat clear command: adb shell logcat -v threadtime -b all -c.
Launch the app: adb shell am start <app_pkg>/<app_main_activity>.
Run the logcat command: adb shell logcat -v threadtime -b all -d and capture the cool_app_launch_time timer value.
After the app has fully launched, force stop the app: adb shell am force-stop <app_pkg>.
Repeat the steps for the number of iterations required.

Cool iterations

Use the following code sample to execute the latency cool launch test and print the latency values for the requested iterations.

Java
Kotlin

Java

for (int i = 0; i < iterations; i++) {
   if (!launchAppUsingMonkey(DSN, appPackage)) 
       launchAppUsingIntent(DSN, appIntent);
   Thread.sleep(10);
                
   BufferedReader read = new BufferedReader
       (new InputStreamReader(Runtime.getRuntime()
        .exec("adb -s "+ DSN +" shell logcat -v threadtime -b all -d")
        .getInputStream()));
                
   String line_CoolApp;
   String line_CoolActivity;

    log_file_writer(read);
   File adb_log_file = new File( kpi_log_file_path + "/KPI_Log.txt");
   
   if (deviceType == "Tablet") {
       timer = get_vitals_timer(adb_log_file, appPackage, COOL_APP); 
       
   } else if (deviceType == "FTV") {
      timer = get_displayed_timer(adb_log_file, appPackage);
      
      if (timer == 0) {
        timer = get_vitals_timer(adb_log_file, appPackage, COOL_APP);
      }
   }
   
   if (timer == 0) {
      timer = get_vitals_timer(adb_log_file, appPackage, COOL_ACTIVITY); 
   }

   forceStopApp(DSN, appPackage);
   Thread.sleep(10);
   Runtime.getRuntime().exec("adb -s "+ DSN +" shell logcat -b all -c");
}

Kotlin

for (int i = 0; i < iterations; i++) {
    if (!launchAppUsingMonkey(DSN, appPackage)) {
        launchAppUsingIntent(DSN, appIntent)
    }
    Thread.sleep(10)

    val read = BufferedReader(InputStreamReader(Runtime.getRuntime().exec("adb -s ${DSN} shell logcat -v threadtime -b all -d").getInputStream()))

    val line_CoolApp = read.readLine()
    val line_CoolActivity = read.readLine()

    log_file_writer(read)
    val adb_log_file = File(kpi_log_file_path + "/KPI_Log.txt")

    when (deviceType) {
        "Tablet" -> timer = get_vitals_timer(adb_log_file, appPackage, COOL_APP)
        "FTV" -> timer = get_displayed_timer(adb_log_file, appPackage)
                .takeIf { it != 0 }
                ?: get_vitals_timer(adb_log_file, appPackage, COOL_APP)
    }

    if (timer == 0) {
        timer = get_vitals_timer(adb_log_file, appPackage, COOL_ACTIVITY)
    }

    forceStopApp(DSN, appPackage)
    Thread.sleep(10)
    Runtime.getRuntime().exec("adb -s ${DSN} shell logcat -b all -c")
}

Device cool vital logs

The following is an example of how the device cool vital logs will appear:

03-03 12:42:32.589   892   992 I Vlog    : PhoneWindowManager:ScreenTime:fgtracking=false;DV;1,Timer=1.0;TI;1,unit=count;DV;1,metadata=!{"d"#{"groupId"#"<APP_PACKAGE_NAME>"$"schemaId"#"123"$"startTimeMs"#"1.677827544773E12"$"packageName"#"<APP_PACKAGE_NAME>"$"endTimeMs"#"1.677827552587E12"$"durationMs"#"7813.0"}};DV;1:HI
03-03 12:42:33.657   892  1092 I Vlog    : performance:cool_app_launch_time:fgtracking=false;DV;1,key=<APP_PACKAGE_NAME>;DV;1,**Timer****=****1333.0**;TI;1,unit=ms;DV;1,metadata=App_Metadata!{"d"#{"groupId"#"abc"$"schemaId"#"abc"$"app_version"#"123"$"isChild"#"false"}$"m"#{"prev"#"<APP_PACKAGE_NAME>"$"context"#"abc"}};DV;1:HI
03-03 12:42:33.661   892  1092 I Vlog    : performance:user_app_launch_cool:fgtracking=false;DV;1,key=<APP_PACKAGE_NAME>;DV;1,Counter=1;CT;1,unit=count;DV;1,metadata=App_Metadata!{"d"#{"groupId"#"abc"$"schemaId"#"abc"$"isChild"#"false"}};DV;1:HI
03-03 12:49:20.880   892  1092 I Vlog    : performance:cool_app_launch_time:fgtracking=false;DV;1,key=<APP_PACKAGE_NAME>;DV;1,**Timer****=****1225.0**;TI;1,unit=ms;DV;1,metadata=App_Metadata!{"d"#{"groupId"#"abc"$"schemaId"#"abc"$"app_version"#"123"$"isChild"#"false"}$"m"#{"prev"#"<APP_PACKAGE_NAME>"$"context"#"abc"}};DV;1:HI
03-03 12:49:20.918   892  1092 I Vlog    : performance:user_app_launch_cool:fgtracking=false;DV;1,key=<APP_PACKAGE_NAME>;DV;1,Counter=1;CT;1,unit=count;DV;1,metadata=App_Metadata!{"d"#{"groupId"#"abc"$"schemaId"#"abc"$"isChild"#"false"}};DV;1:HI

Warm launch until first frame

Latency warm launch until first frame consists of launching the app from the background or from recent apps and measuring the time taken to draw the first frame. This metric is used to measure the performance of the app because the app typically takes less time to launch when launching from background as required services are already available in the cache.

To run measure the latency warm launch until first frame:

Make sure the app is running in the background.
Run the logcat clear command: adb shell logcat -v threadtime -b all -c.
Launch the app: adb shell am start <app_pkg>/<app_main_activity>.
Run the logcat command: adb shell logcat -v threadtime -b all -d and capture the warm_app_warm_transition_launch_time timer value.
After the app has fully launched, run: adb shell input keyevent KEYCODE_HOME.
Repeat the steps for the number of iterations required.

Warm iterations

Use the following code sample to run the latency warm launch test and print the latency values for the requested iterations.

Java
Kotlin

Java

for (int i = 0; i < iterations; i++) {
   if (!launchAppUsingMonkey(DSN, appPackage)) 
       launchAppUsingIntent(DSN, appIntent);
   Thread.sleep(10);
                
   BufferedReader read = new BufferedReader
       (new InputStreamReader(Runtime.getRuntime()
        .exec("adb -s "+ DSN +" shell logcat -v threadtime -b all -d")
        .getInputStream()));
                
   String line_CoolApp;
   String line_CoolActivity;

   log_file_writer(read);
   String adb_log_file = kpi_log_file_path + "/KPI_Log.txt";
   
   if (deviceType == "Tablet") {
       timer = get_vitals_timer(adb_log_file, appPackage, WARM_APP_WARM); 
       
   } else if (deviceType == "FTV") {
      timer = get_displayed_timer(adb_log_file, appPackage);
      
      if (timer == 0) {
        timer = get_vitals_timer(adb_log_file, appPackage, WARM_APP_WARM);
      }
   }
   
   if (timer == 0) {
      timer = get_vitals_timer(adb_log_file, appPackage, WARM_APP_COOL); 
   }

   Runtime.getRuntime().exec("adb -s "+ DSN +" shell input keyevent KEYCODE_HOME");
   Thread.sleep(10);
   Runtime.getRuntime().exec("adb -s "+ DSN +" shell logcat -b all -c");
}

Kotlin

for (int i = 0; i < iterations; i++) {
    if (!launchAppUsingMonkey(DSN, appPackage)) 
        launchAppUsingIntent(DSN, appIntent)
    Thread.sleep(10)

    val read = BufferedReader(InputStreamReader(Runtime.getRuntime().exec("adb -s ${DSN} shell logcat -v threadtime -b all -d").getInputStream()))

    val line_CoolApp = read.readLine()
    val line_CoolActivity = read.readLine()

    log_file_writer(read)
    val adb_log_file = kpi_log_file_path + "/KPI_Log.txt"

    when (deviceType) {
        "Tablet" -> timer = get_vitals_timer(adb_log_file, appPackage, WARM_APP_WARM)
        "FTV" -> timer = get_displayed_timer(adb_log_file, appPackage)
            .takeIf { it != 0 } ?: get_vitals_timer(adb_log_file, appPackage, WARM_APP_WARM)
    }

    if (timer == 0) {
        timer = get_vitals_timer(adb_log_file, appPackage, WARM_APP_COOL)
    }

    Runtime.getRuntime().exec("adb -s ${DSN} shell input keyevent KEYCODE_HOME")
    Thread.sleep(10)
    Runtime.getRuntime().exec("adb -s ${DSN} shell logcat -b all -c")
}

Device warm vital logs

The following is an example of how the device warm vital logs will appear:

03-03 12:51:16.367   892  1066 I Vlog    : Thermal:ScreenOn_SensorThrottling_P2P_Off:fgtracking=false;DV;1,key=0;DV;1,Timer=8.072222222222222E-4;TI;1,unit=hours;DV;1,metadata=<APP_PACKAGE_NAME>!{"d"#{"groupId"#"123"$"schemaId"#"123"$"zone"#"soc"}};DV;1:HI
03-03 12:51:16.367   892  1066 I Vlog    : Thermal:ScreenOn_SensorThrottling_P2P_Off:fgtracking=false;DV;1,key=0;DV;1,Timer=8.072222222222222E-4;TI;1,unit=hours;DV;1,metadata=<APP_PACKAGE_NAME>!{"d"#{"groupId"#"123"$"schemaId"#"123"$"zone"#"bottom"}};DV;1:HI
03-03 12:51:16.367   892  1066 I Vlog    : Thermal:ScreenOn_SensorThrottling_P2P_Off:fgtracking=false;DV;1,key=0;DV;1,Timer=8.072222222222222E-4;TI;1,unit=hours;DV;1,metadata=<APP_PACKAGE_NAME>!{"d"#{"groupId"#"123"$"schemaId"#"123"$"zone"#"side"}};DV;1:HI
03-03 12:51:16.368   892  1066 I Vlog    : Thermal:Screen_On_Thermal_Throttling_P2P_Off:fgtracking=false;DV;1,key=0;DV;1,Timer=8.075E-4;TI;1,unit=hours;DV;1,metadata=<APP_PACKAGE_NAME>!{"d"#{"groupId"#"123"$"schemaId"#"abc"}};DV;1:HI
03-03 12:51:16.384   892  1092 I Vlog    : performance:warm_app_warm_transition_launch_time:fgtracking=false;DV;1,key=<APP_PACKAGE_NAME>;DV;1,**Timer****=****191.0**;TI;1,unit=ms;DV;1,metadata=<APP_PACKAGE_NAME>!{"d"#{"groupId"#"123"$"schemaId"#"abc"$"app_version"#"123"$"isChild"#"false"}$"m"#{"prev"#"<APP_PACKAGE_NAME>"$"context"#"1234"}};DV;1:HI
03-03 12:51:16.395   892  1092 I Vlog    : performance:user_app_launch_warm:fgtracking=false;DV;1,key=<APP_PACKAGE_NAME>;DV;1,Counter=1;CT;1,unit=count;DV;1,metadata=<APP_PACKAGE_NAME>!{"d"#{"groupId"#"123"$"schemaId"#"abc"$"isChild"#"false"}};DV;1:HI

Ready-to-use

The ready-to-use (RTU) KPI is measured as the time taken by an app from launch to the ready to use state (e.g., sign in page or home page of an app). RTU is used to measure the app performance while launching the app and helps identify issues in the app. This KPI replicates the customer use case and provides the best experience to the end customer.

RTU before sign in means measuring the RTU screen before sign in. While performing before sign in cool or warm launches, make sure that the app is not logged in.
RTU after sign in means measuring the RTU screen after sign in. While performing after sign in cool or warm launches, make sure that the app is logged in using test credentials.

RTU before sign in cool launch is launching the app from home screen or launching the app after a force stop and then measuring the time taken to draw the RTU state of the app or draw the sign in page of the app.

Run the logcat clear command: adb shell logcat -v threadtime -b all -c.
Launch the app: adb shell am start <app_pkg>/<app_main_activity>.
Run the logcat command: adb shell logcat -v threadtime -b all -d and capture the fully_drawn timer value.
After the app has fully launched and timer value is captured, use the following command to force stop the app: adb shell am force-stop <app_pkg>.
Repeat the steps for the number of required iterations.

RTU before sign in warm launch is launching the app from the background or from recent apps and then measuring the time taken to draw the RTU state of the app or draw the sign in page of the app.

Make sure the app is running in background.
Run the logcat clear command: adb shell logcat -v threadtime -b all -c.
Launch the app: adb shell am start <app_pkg>/<app_main_activity>.
Run the logcat command: adb shell logcat -v threadtime -b all -d and capture the fully_drawn timer value.
After the app has fully launched, use the following command to move the app to the background: adb shell input keyevent KEYCODE_HOME.
Repeat the steps for the number of required iterations.

RTU after sign in cool launch is launching the app from home page or launching the app after force stop and then measuring the time taken to draw the RTU state of the app after sign in or drawing the home page of the app.

Make sure the app is signed in with test credentials.
Run the logcat clear command: adb shell logcat -v threadtime -b all -c.
Launch the app: adb shell am start <app_pkg>/<app_main_activity>.
Run the logcat command and capture the fully-drawn timer value: adb shell logcat -v threadtime -b all -d.
After the app has fully launched and timer value has been captured, run the command to force stop the app: adb shell am force-stop <app_pkg>.
Repeat the steps for the number of required iterations.

RTU after sign in warm launch is launching the app from background or from recent apps and then measuring the time taken to draw the RTU state of the app after sign in or draw the home page of the app.

Make sure the app is running in background and is logged in with test credentials.
Run the logcat clear command: adb shell logcat -v threadtime -b all -c.
Launch the app: adb shell am start <app_pkg>/<app_main_activity>.
Run the logcat command and capture the fully-drawn timer value: adb shell logcat -v threadtime -b all -d.
After the app has fully launched, run the command to move the app to background: adb shell input keyevent KEYCODE_HOME.
Repeat the steps for the number of required iterations.

RTU ADB logs

Activity fully_drawn log line for Monster Strike app on Amazon Fire tablet device:

**`10`****`-`****`06`****` `****`10`****`:`****`28`****`:`****`03.932`****` `****`678`****` `****`703`****` I `****`ActivityTaskManager`****`:`****` `****`Fully`****` drawn `****`<APP_PACKAGE_NAME`****`>`****`:`****` `****`+`****`5s36ms`**

Activity fully_drawn log line for Peacock TV app on Amazon Fire TV device:

03-27 13:19:26.362   527   537 I am_activity_fully_drawn_time: [0,180804427,<APP_MAIN_ACTIVITY>,7949,7859]
**03****-****27** **** **13****:****19****:****26.362** **** **527** **** **537** **I** **ActivityManager****:** **** **Fully** **drawn** **<****APP_MAIN_ACTIVITY****>****:** **** **+****7s949ms** **** **(****total** **+****7s859ms****)**
03-27 13:19:26.362   527   537 I sysui_multi_action: [324,0,757,1090,758,12,806,<APP_PACKAGE_NAME>,871,<APP_MAIN_ACTIVITY>,1091,7998]

Activity fully_drawn log line for Max app on Amazon Fire TV device:

03-27 14:26:57.063   527  1134 I am_activity_fully_drawn_time: [0,170114187,<APP_PACKAGE_NAME>,3588,3462]
**03****-****27** **** **14****:****26****:****57.063** **** **527** **** **1134** **I** **ActivityManager****:** **** **Fully** **drawn <APP_MAIN_ACTIVITY>****:** **** **+****3s588ms** **** **(****total** **+****3s462ms****)**

Memory KPI

The Memory KPI provides a detailed overview of the memory consumption and represents the memory consumption or the memory consumption along with other memory values by the foreground activity, as well as the background activity of the app. Along with the memory values, this KPI also measures foreground or background CPU usage, RAM usage, RAM free, and other specifics.

Foreground memory

The foreground memory KPI captures the memory consumed by the app in the foreground, for this the app video or game is made to play for 15 minutes in the foreground and then the memory consumption by the app is fetched and calculated.

Calculating the foreground memory consumption:

Run the logcat clear command: adb shell logcat -v threadtime -b all -c.
Launch the app: adb shell am start <app_pkg>/<app_main_activity>.
In the app, sign in and the play the app’s core contents (game or video).
Run the dumpsys command and capture the total_pss memory value: adb shell dumpsys meminfo -a <app_pkg>.
After the memory value is captured, run the command to force stop the app: adb shell am force-stop <app_pkg>.
Repeat the steps for the number of required iterations.

Background memory

The background memory KPI captures the memory consumed by the app in the background, for this the app video or game is made to play for 15 minutes in the foreground, moved to the background, and then the memory consumption by the app is fetched and calculated.

Calculate the background memory consumption:

Run the logcat clear command: adb shell logcat -v threadtime -b all -c.
Launch the app: adb shell am start <app_pkg>/<app_main_activity>.
In the app, sign in and the play the app’s core contents (game or video).
After the app navigations are complete, run the command to move the app to the background: adb shell input keyevent KEYCODE_HOME.
Wait 10 - 15 seconds for the app to stabilize in the background.
Run the dumpsys command and capture the total_pss memory value: adb shell dumpsys meminfo -a <app_pkg>.
Repeat the steps for the number of required iterations.

Memory ADB dump log

The Proportional Set Size (PSS) total is the amount of memory consumed by the app on the device. PSS total is used to calculate the memory consumption of app when it is in the foreground or background.

                 Pss      Pss   Shared  Private   Shared  Private  SwapPss     Heap     Heap     Heap
                Total    Clean    Dirty    Dirty    Clean    Clean    Dirty     Size    Alloc     Free
                                   
 Native Heap   115268        0      384   115020      100      208       22   151552   119143    32408
 Dalvik Heap    15846        0      264    15124      140      676       11    21026    14882     6144
Dalvik Other     8864        0       40     8864        0        0        0                           
       Stack      136        0        4      136        0        0        0                           
      Ashmem      132        0      264        0       12        0        0                           
   Other dev       48        0      156        0        0       48        0                           
    .so mmap    15819     9796      656      596    26112     9796       20                           
   .apk mmap     2103      432        0        0    26868      432        0                           
   .dex mmap    39396    37468        0        4    17052    37468        0                           
   .oat mmap     1592      452        0        0    13724      452        0                           
   .art mmap     2699      304      808     1956    12044      304        0                           
  Other mmap      274        0       12        4      636      244        0                           
   GL mtrack    42152        0        0    42152        0        0        0                           
     Unknown     2695        0       92     2684       60        0        0                           
       TOTAL   **247077**    48452     2680   186540    96748    49628       53   172578   134025    38552

Additional Links

For more useful ADB commands, see ADB command list. For more information, see Test criteria group 2: App behavior on Fire TV device

Writing a KPI log file

Use the following code sample to write a KPI log file for one iteration.

Note: This will store and write the logs for one iteration. To write the entire test course for all iterations, make sure the new file and filewriter objects are not created and remember not to close the filewriter object.

Java
Kotlin

Java

public void log_file_writer(BufferedReader bf_read) {
 File adb_log_file = new File(kpi_log_file_path + "/KPI_Log.txt");
 FileWriter fileWriter = new FileWriter(adb_log_file);
 try {
    String reader = null;
     while ((reader = bf_read.readLine()) != null) {
            fileWriter.write(reader.trim() + "\n");
     }
  }
  catch (Exception e) {
     System.out.println(e);
  } 
  finally {
     fileWriter.flush();
     fileWriter.close();
     bf_read.close();
  }   
}

Kotlin

import java.io.File
import java.io.FileWriter
import java.io.BufferedReader


fun logFileWriter(bfRead: BufferedReader) {
    val adbLogFile = File("$kpi_log_file_path/KPI_Log.txt")
    val fileWriter = FileWriter(adbLogFile)
    try {
        var reader: String?
        while (bfRead.readLine().also { reader = it } != null) {
            fileWriter.write(reader?.trim() + "\n")
        }
    } catch (e: Exception) {
        println(e)
    } finally {
        fileWriter.flush()
        fileWriter.close()
        bfRead.close()
    }
}

Last updated: Dec 20, 2023

App Performance Scripts

Setup

Test strategy

Get device type

Generating main launcher activity of the app

Launching an app with the main launcher activity (app intent)

Launching an app using the Monkey tool

Force stopping an app

Latency KPI

Common Abbreviations used in Logs

General ADB logcat commands used

Capture timer values from vital buffer logs

Capture timer values from activity manager buffer logs

Cool launch until first frame

Cool iterations

Device cool vital logs

Warm launch until first frame

Warm iterations

Device warm vital logs

Ready-to-use

Steps to measure RTU before sign in cool launch

Steps to measure RTU before sign in warm launch

Steps to measure RTU after sign in cool launch

Steps to measure RTU after sign in warm launch

RTU ADB logs

Memory KPI

Foreground memory

Background memory

Memory ADB dump log

Additional Links

Writing a KPI log file