Using Image Recognition for UI Automation with Sikuli: Principles, Functions, and Code Examples

When discussing UI automation, most people first think of element‑based methods such as XPath, ID, or CSS selectors, but many scenarios—especially on web or mobile platforms—require locating elements based on visual content, which traditional selectors cannot achieve. This article introduces the use of image‑recognition technology in testing.

Typical image‑recognition testing scenarios include:

Capturing screenshots of the application under test and using recognition algorithms to detect predefined interactive controls, triggering actions automatically.

Validating test results by matching screenshots of the current UI against expected images.

Performing performance testing, such as measuring app response times, through visual comparison.

Principle

Sikuli scripts are written in Jython and simulate keyboard and mouse events via image recognition. The core consists of a Java library with two parts: a java.awt.Robot component that sends input events to screen coordinates identified by a C++ engine (built on OpenCV) searching for target images, and a higher‑level Java/Jython API that provides simple commands for script authors.

Key Functions

Below are the most commonly used Sikuli functions (each followed by a brief description and a placeholder for example code):

find(x) – Locate a single occurrence of image x on the screen.

findall(x) – Locate all occurrences of image x on the screen.

wait(x, 10) – Wait up to 10 seconds for image x to appear.

waitVanish(x, 10) – Wait up to 10 seconds for image x to disappear.

exists(x) – Check whether image x exists; returns None if not found.

click(x) – Left‑click the best‑matched GUI component for image x .

doubleClick(x) – Double‑click the best‑matched component.

rightClick() – Right‑click the best‑matched component.

hover(x) – Move the mouse cursor over the best‑matched component.

dragDrop(x, y) – Drag image x and drop it onto image y .

type(x, "text") – Type the given text into the focused element identified by x .

paste(x, "text") – Paste the given text into the focused element.

Each function is typically illustrated with a short code example (omitted here for brevity).

Advantages

The main benefits of using Sikuli’s image‑recognition approach are:

Simple, low‑learning‑curve code – a screenshot is often enough to start automating.

Effective for applications with custom or game‑style UI elements that lack standard DOM hooks.

Reusable, open‑source library that can be extended.

Ability to automate visual components such as Flash that are not accessible via traditional selectors.

Limitations

However, the method also has drawbacks:

Screen must be unobstructed; any overlay prevents recognition.

Screen resolution changes require new reference images.

Tests must run in the foreground; background execution is not supported.

For further details and a complete demo, refer to the Sikuli productivity page linked in the original article.