Applying Deep Learning for Automated UI Bug Detection in Mobile Apps

Background Overview Manual UI testing is costly and inefficient, especially with the explosion of device models and OS versions. Simple visual bugs (e.g., overlapping text, misaligned controls) are hard to capture with traditional assertion‑based automation.

Practice Scheme The authors propose a computer‑vision approach built on the open‑source PaddlePaddle framework. By collecting screenshots from multiple devices, they perform horizontal comparison using reference devices and image‑matching techniques such as SIFT, SURF, histogram similarity, and pHash to flag potential compatibility issues.

Data Preparation Because deep learning requires large datasets, they generate training samples automatically. For keyboard‑style bugs they decompose UI components into three parts (①②③) and recombine them, inserting blank regions to create negative samples. For less structured bugs (e.g., text overlap) they apply data‑augmentation methods like random translation, rotation, and noise injection.

Model Definition A convolutional neural network (CNN) is employed, consisting of fully‑connected layers, activation functions, convolution, pooling, and a Softmax classifier. Input images are resized to 3‑channel 100×100 tensors, and the model predicts one of three UI‑bug categories: keyboard style, text overlap, or page blank.

Training & Validation The model was trained on tens of thousands of synthetic samples and evaluated on a held‑out test set (1,000 images per class). Reported accuracies are 97% overall, with 95% for keyboard‑style bugs and 77% for text‑overlap bugs. Errors often stem from improper cropping of overlapping text regions, suggesting future improvements in preprocessing.

PaddlePaddle Features PaddlePaddle supports Linux, macOS, and Docker, offers easy environment setup, fast execution, distributed training, and a rich set of demos (image classification, word‑vector analysis, recommendation, sentiment analysis, etc.). Compared with TensorFlow and Caffe, it provides specific advantages for this use case, as illustrated in the accompanying table.

Conclusion The practice validates that deep‑learning models can automate UI‑bug detection, reducing reliance on manual visual checks. The approach is generic across apps, having processed 470 images with 10,842 verification runs and identified 13 effective bugs. Future work will expand the number of supported bug categories while maintaining high recall and precision.