Automated Captcha Recognition Using Machine Learning

This article explores how to efficiently recognize login page captchas during automated testing, aiming to improve test efficiency.

For simple drag‑and‑drop captchas, Selenium's drag_and_drop function can be used to move elements and verify that the visitor is not a robot.

When dealing with image captchas, the author proposes a machine‑learning pipeline: first segment the image to isolate each character, then classify each segment using a supervised learning model.

The human approach to captcha solving is described as (1) locating the N characters in the image, and (2) recognizing each character to obtain the final code.

Image segmentation is presented as the process of splitting a multi‑character image into single‑character sub‑images, analogous to dynamic programming splitting a large problem into smaller sub‑problems.

Classification is framed as a supervised learning problem. The author contrasts regression (continuous output) with classification (discrete output) and gives a house‑price prediction example to illustrate the concept.

To create a captcha dataset, the steps include randomly selecting characters, placing them on a background, adding noise, applying random scaling/rotation/projection, and finally storing both the images and their corresponding labels (e.g., in CSV files). The Python captcha library can generate such images with a few lines of code.

After generation, images are converted to grayscale to simplify processing. Thresholding (binarization) separates foreground characters from background noise, and simple pixel‑neighbour analysis can further remove isolated noise points.

For segmentation, several clustering algorithms are discussed: K‑means (producing circular clusters), EM‑GMM (elliptical clusters via Expectation‑Maximization), Mean‑Shift (gradient‑based mode seeking), and the watershed algorithm (interpreted as “rainfall” on a topographic surface).

The article reviews classic classification algorithms such as logistic regression, support vector machines (with kernel tricks), K‑nearest neighbors, decision trees, random forests (bagging), and boosting, highlighting their strengths and limitations for captcha recognition.

Neural networks, particularly convolutional neural networks (CNNs), are presented as a modern solution. Both multi‑class and multi‑label formulations are described, with details on common components (optimizers like Adam, pooling, dropout, activation functions). Training results show around 93.9% accuracy on a test set, with early stopping to prevent over‑fitting.

In conclusion, the author notes that captchas are becoming increasingly complex (e.g., image selection challenges) and suggests that, when possible, collaborating with developers to bypass captchas may be more practical than attempting perfect automated recognition.