Structured Layout Information and Guided Line Method for UI Component Detection

This article introduces a structured approach to processing layout information in UI images, aiming to improve code reuse and reduce redundant code generation in UI2CODE projects.

Two main problems are identified: (1) recognizing business components that appear across multiple pages, and (2) detecting duplicate layouts such as repeated GridView items.

For problem (1), the traditional object‑detection (SSD) solution requires massive labeled data and suffers from training difficulty and uncontrollable results.

The proposed solution consists of two steps: (1) layout information structuring, and (2) layout comparison.

In the structuring phase, each pair of controls is represented by a Connection object that records the attributes of both controls (type, position, size) and one or more virtual lines describing direction, distance, and alignment. A layout is then expressed as a set of Connections.

In the comparison phase, similar layouts are found by matching Connections based on attribute similarity, cosine similarity of line vectors, and custom rules. The “guided line method” introduces a Leader‑Follower mechanism: a Leader path is selected, and a Follower attempts to follow the same sequence of Connections. When the Follower can no longer keep up, the traversed paths represent a pair of similar layouts.

The method is applied to both business component identification and duplicate‑layout detection, demonstrating that once control information is structured, new components can be matched quickly without retraining models.

Experimental results show that the algorithm can reliably locate repeated items (e.g., eight similar GridView cells) and significantly improve the usability of generated code.