Choosing a Web Crawler: Nutch, Crawler4j, WebMagic, WebCollector, Scrapy, or Others

When developing a web crawler you can choose among Nutch, Crawler4j, WebMagic, WebCollector, Scrapy, or other tools. Based on experience they can be roughly divided into three categories: distributed crawlers, Java single‑machine crawlers, and non‑Java single‑machine crawlers.

First category – Distributed crawlers (Nutch) : Nutch is popular but generally a poor choice for most users because it is designed for search engines, wastes resources on unnecessary processing, depends on Hadoop which can slow down crawling on small clusters, has a cumbersome plugin system, and requires extensive code changes for precise data extraction.

Second category – Java single‑machine crawlers : Java offers a mature ecosystem with frameworks such as Crawler4j, WebMagic, and WebCollector. Important concerns include multi‑threading, proxy support, duplicate‑URL filtering, JavaScript‑generated content (handled via headless browsers like HtmlUnit or Selenium), AJAX handling, login via cookies, extraction using CSS selectors or XPath, persistence via pipelines or custom database code, and dealing with site blocking by rotating proxies. Performance largely depends on thread count, network speed, and persistence logic.

Third category – Non‑Java single‑machine crawlers : Python (e.g., Scrapy) can achieve the same tasks with fewer lines of code but may require more testing for stability; C++ has a steep learning curve; Ruby and PHP are suitable for small‑scale tasks but suffer from limited community support and potential bugs.

Source: 36大数据 (36dsj.com).