Automated Mobile UI Test Script Recording and Playback System

1. Requirement Background

Current mobile UI testing platforms require testers to manually write test case scripts in various languages (e.g., Lua, Java) before running them, which is difficult and costly.

2. Desired Effect

By recording the screen, the user's operations on the phone are captured and automatically converted into executable test scripts, addressing the drawbacks of manual scripting.

We have implemented automatic script recording that generates Python test cases.

3. Usage Introduction

1. Platform support: http://opentest.360.cn/phonerenter.html

2. Prerequisite: Open a cloud device on the cloud real‑device platform and select "Script Recording".

3. Steps:

(1) Upload APK (select the APK to be tested and upload it)

(2) View control information (hover over a UI element on the screen to see its details)

(3) Record script (record mouse actions on the screen; add actions that are not reachable)

(4) Script playback (play back the script to verify execution; modify parameters if needed)

(5) Save script (after successful execution, save it to the user's script list)

4. Technical Implementation

1. Upload APK

Frontend upload, node machine download

adb push … adb pm clear packagename # clear historical data adb pm uninstall packagename # uninstall previous APK adb pm install -r apkpath # install APK adb am start -n packagename/activityname # launch app

2. View Control Information

Visual tool: UI Automator Viewer – Feature Overview

Control Information – Feature Display

Control Information – Technical Implementation

First Version

Command used:

adb shell uiautomator dump

Second Version

Because the first version’s dump operation was slow, we switched to using AccessibilityService to obtain the window’s root node and process the UI hierarchy into XML.

AccessibilityService is an Android service that provides accessibility assistance; it can monitor UI events, retrieve the current window, and inspect node information such as md5, index, resource-id, class, text, etc.

Backend Service Implementation

We create a custom service class (e.g., AccessibilitySampleService ) that extends AccessibilityService , overrides onAccessibilityEvent and onInterrupt , registers it in AndroidManifest.xml , and processes events to locate and interact with UI nodes.

To map mouse coordinates from the web page to the device screen, we convert the XML data to a DOM tree, match the coordinates, and select the smallest-area control containing the point.

5. Script Recording

Recording user operations on the phone UI

Supports 12 actions: back, home, menu, click by md5, find by md5, click by text, find by text, swipe, drag, send text, press backspace, sleep (calculate interval between actions).

Problems Encountered

(1) Distinguishing click, swipe, drag: mouse events down, move, up, leave

Solution:

The diagram shows the classification:

A. Use a distance threshold to decide click vs. swipe/drag; small movement = click, larger = swipe/drag.

B. Distinguish swipe from drag by the press duration: long press = drag, short press = swipe.

When clicking, if the control has text we click by text; otherwise we compute a unique md5 based on its position in the XML tree.

(2) Handling device rotation and screen‑to‑control scaling

Solution: Monitor device rotation in the background; if rotated, adjust coordinates before converting to pixel values.

6. Script Playback

Execute actions based on saved information

7. Save Script

After successful playback, the script is saved and a Python script is generated, visible in the script management page for further execution.

8. Summary

1) The dump operation needed optimization for real‑time requirements. 2) Precise gesture recording requires careful threshold tuning to achieve high accuracy for most user habits.