Introduction to Search Engine Technology and Information Retrieval

Search engines combine natural language processing, information retrieval, web architecture, and distributed data processing to help users accurately locate information.

The current market is dominated by major portals, but search is shifting toward specialized vertical searches (e.g., Zhihu, DeWu, Meituan).

Information retrieval is defined as finding unstructured textual material that satisfies an information need within a large collection (Manning, 2008). Two fundamental performance metrics are recall and precision, which often trade off against each other.

This article introduces the basic workflow of search engines, focusing on document structure, inverted index, query operators, and ranking algorithms.

Document structure is hierarchical: metadata (URL, keywords, author, date), body (title, main content), and external information (links). Different sections contain varying amounts of information; for example, titles carry higher information density than body paragraphs.

Search engines handle two types of documents: web‑level and enterprise‑level. Understanding and storing the corpus requires parsing this hierarchy.

Inverted indexes are the core storage mechanism. Two main layouts are discussed: a flat (horizontal) layout where each block is independent, and a vertical layout that preserves hierarchical relationships (document → block → paragraph → sentence). The vertical layout enables more precise queries on specific positions.

Query operators are categorized into three groups: operators that build new inverted indexes (#SYN, #NEAR, #WINDOW), operators that generate score lists (#SCORE), and operators that merge score lists (#AND, #OR, #WSUM). These operators allow construction of complex queries.

Ranking algorithms range from simple Boolean models (UnrankedBoolean, RankedBoolean) to probabilistic models such as BM25, which adjusts term frequency, document length, and user weight via parameters K1, B, and K3. Smoothing techniques like Jelinek‑Mercer and Dirichlet prior are used to handle rare terms and short documents.

Advanced models such as Indri combine statistical language models with Bayesian networks, treating queries as structured networks that are matched against document networks to compute likelihood scores.

Overall, the article provides a foundational overview of search engine components, preparing readers to explore deeper algorithmic implementations.