Understanding Anti‑Crawling: Definitions, Current Landscape, Classifications, and Strategic Insights

Personal introduction: Pan Hongmin joined Qunar in July 2015 and now works in the large‑accommodation division’s front‑end team, focusing on crawler analysis, anti‑crawling strategy development, platform maintenance, data analysis, and data cleaning.

The previous article covered crawlers; this one focuses on anti‑crawling. Anti‑crawling is defined as the technology that identifies crawlers through unique characteristics, emphasizing accurate distinction between bots and legitimate users rather than post‑detection measures like captchas or forced logins.

Anti‑crawling techniques evolve alongside crawler advancements. Simple crawlers can be detected via user‑agent strings or IP frequency, but more sophisticated bots employ low‑cost disguises.

Current anti‑crawling literature is scarce, often describing only basic detection methods. No universal, permanent solution exists; companies protect their own strategies and keep them confidential.

Anti‑crawling consists of two main processes: crawler identification (the core and most challenging phase) and data protection (safeguarding responses sent to clients).

Based on different criteria, anti‑crawling can be classified as client‑side, middle‑layer, or server‑side, and further as real‑time or non‑real‑time. Client‑side detection runs in front‑end scripts but is limited because bots can bypass scripts. Middle‑layer detection inserts an interception layer between client and server, acting as a filter that reduces server load and serves multiple services. Server‑side detection occurs entirely on the server, handling only genuine users but requiring separate implementations for each business line.

Real‑time anti‑crawling intercepts bots during access, while non‑real‑time approaches analyze suspected bots first and then block similar patterns later.

The author reflects that anti‑crawling strategies are often judged by detection rate and false‑positive rate, yet the ongoing arms race means strategies have limited lifespans; once crawlers adapt, previously effective measures become obsolete.

There is no everlasting anti‑crawling solution; effectiveness varies with the crawler’s sophistication.

Using a volleyball analogy, the author compares strong attacks, quick attacks, and diversified attacks to anti‑crawling tactics, emphasizing the need for multiple detection points and flexible responses.

Systematizing anti‑crawling is essential: a robust platform should support rapid deployment of strategies, centralize detection, and enable coordinated responses across different services, reducing manual effort and resource consumption.

In summary, anti‑crawling is a long‑term, systematic effort rather than a series of isolated tactics; a dedicated anti‑crawling platform is crucial for an online travel service handling sensitive data.