Intelligent UI Traversal and Automated Testing for JD Mobile Apps and Mini‑Programs

Background – JD's mobile app and mini‑program traffic have surged, leading to a rapid increase in pages and features, which strains manual testing and traditional script‑based automation. A script‑free UI traversal tool that can automatically discover page elements and detect defects is needed for high‑frequency releases.

Stability Testing – Traditional tools like Monkey generate random events, while UI traversal captures actionable elements first, then interacts with them. Two element‑capture approaches are discussed: extracting a GUI tree on Android and using image‑based segmentation with OCR and icon detection for H5, mini‑programs, and iOS.

Technical Practice – The framework integrates dump‑hierarchy view‑tree extraction and OpenCV‑based image segmentation to support multiple platforms. iOS switched from view‑tree to image segmentation due to performance issues.

Intelligent Element Acquisition – Screenshots are sent to JD’s cloud image service, which segments the page, runs AI models to recognize UI components, and returns element coordinates.

Image Segmentation Process – (1) Pre‑process images (crop, grayscale, binarize); (2) Horizontal/vertical split based on pixel patterns; (3) Aggregate UI regions; (4) Detect icons with YOLO and text with OCR, producing the final segmented view.

Dynamic Popup Handling – The service first detects pop‑ups via projection analysis, then resolves them either by OCR‑based keyword matching (for permission dialogs) or by AI‑driven icon detection to click close buttons.

Page Uniqueness Identification – Initially page hashes were used, but false positives occurred. The improved method combines hash similarity with structural contour comparison from segmented images to reliably distinguish pages.

Flexible Page Traversal – Beyond depth‑first search, the tool supports targeted page traversal via URI schemes or mini‑program URL schemes, and allows dynamic parameter configuration to cover personalized page variations.

Rich Assertion Capabilities – Integrated AI services detect UI anomalies such as blank areas, overlapping text, garbled characters, and perform OCR‑based text checks, activity‑based return‑logic validation, and image‑matching for exception handling.

Landing Effect – The tool has been applied to 25 JD apps and mini‑programs, saving significant manual effort and uncovering 56 real issues in two months, with AI‑driven detection accounting for 68% of findings.

Future Outlook – Plans include floor‑level automated testing, reinforcement‑learning‑based exploration, and expanding AI vision capabilities for broader UI monitoring.