AI‑Driven Image Bug Detection in the Xianyu Mobile App

The Xianyu quality team leverages AI to improve app testing by automatically detecting visual bugs through image analysis. With the rise of TensorFlow, they explore how AI can identify page blanks, abnormal controls, and garbled text without deep business knowledge.

Model selection : A simple CNN distinguishes normal from blank‑page screenshots, while an OCR + LSTM pipeline recognises Chinese characters to spot text corruption. Training samples are collected from historical bug screenshots and mock‑generated positive data.

Model re‑training : As the app evolves, new pages cause mis‑classifications. An automated pipeline triggered from the front‑end queues selected images, runs a Jenkins‑scheduled re‑training script, and swaps the old model with the newly trained one, dramatically improving accuracy.

Image handling : Special screenshots that are intentionally blank (e.g., search result pages) are filtered by comparing them against a curated gallery; images with similarity above a threshold are ignored. To reduce visual fatigue, duplicate screenshots are clustered using a bottom‑up hierarchical algorithm.

Implementation (image distance calculation):

def get_pic_array(url, w, h):
    file = cStringIO.StringIO(urllib2.urlopen(url).read())
    img = Image.open(file)  # PIL打开图片
    img = img.resize((w, h))
    try:
        r, g, b, k = img.split()  # rgb通道分离，兼容4通道情况
    except ValueError:
        r, g, b = img.split()
    r_arr = np.array(r).reshape(w * h)
    g_arr = np.array(g).reshape(w * h)
    b_arr = np.array(b).reshape(w * h)
    image_arr = np.concatenate((r_arr, g_arr, b_arr))
    return image_arr

Each image is transformed into a 1‑D vector (w × h × 3) and stacked into an n × (w × h × 3) matrix for clustering.

Clustering : Ward linkage (via linkage(X, 'ward') ) yields the best separation. The critical distance threshold is tuned separately for normal and abnormal images to avoid over‑merging distinct pages.

Summary & outlook : The tool reliably flags whole‑page anomalies and, with ongoing data enrichment, improves text‑error detection. Future work includes integrating LabelImg and training a TensorFlow SSD model to locate widget‑level defects and layout issues.