Web Data Mining and Page Analysis Techniques for Search Engines

The presentation introduces the three core components of a search engine: understanding user queries, crawling and analyzing web data, and linking user behavior with extracted page features to generate relevant results.

It focuses on the second component, detailing the spider system that crawls billions of links daily, stores massive amounts of page and link data, and feeds downloaded pages into a data mining pipeline for content extraction.

Web page analysis begins with building an HTML tree, identifying nodes such as tags, text, titles, and hyperlinks, and extracting hundreds of page attributes (e.g., navigation, title, timestamp, main image) using techniques like classification, clustering, regression, NLP, and topic modeling.

A layout tree is derived from the HTML tree by aggregating node statistics (coordinates, size, style) and simplifying the structure through hierarchical traversal, node deletion, and compression, enabling the division of a page into meaningful regions for deeper semantic understanding.

Region division employs rule‑based or machine‑learning models to decide whether nodes belong to containers such as headers, footers, or content blocks, based on features like area, aspect ratio, and semantic cues.

Finally, the extracted features are used for web page classification, where supervised models (e.g., random forest, logistic regression) predict categories such as news, sports, or e‑commerce, often by stacking multiple binary classifiers for modular and updatable pipelines.

The session concludes with a summary of the page‑level analysis workflow and an invitation to download supplementary PPT materials.