TurboSearch: Tencent AI Lab's Next-Generation Large-Scale Search System

Introduction

Tencent AI Lab has been committed to the development, optimization, and operation of large-scale full-text search engines, accumulating deep technical expertise in massive data search systems and NLP capabilities such as Query analysis and retrieval ranking. Over the past year, based on the original search system, the team has reconstructed a new generation search system called TurboSearch, which inherits the capabilities of the original system in massive indexing and operations while reducing overall complexity, providing better usability, scalability, and private deployment capabilities.

Core Features

TurboSearch offers the following key capabilities: complete distributed massive search system and operation solutions; convenient private deployment support; high-performance indexing and parallel retrieval; multi-granularity indexing and retrieval; support for regular search, category search, WAND, and fine-grained retrieval layer filtering logic; decoupled core components supporting horizontal scaling and customizable capabilities; seamless integration with Tencent AI Lab's NLP capabilities covering Query analysis and ranking; and extensibility to multi-modal vector search scenarios.

Engine Framework

The TurboSearch engine comprises six core capabilities: Search Core Components (basic core capability abstraction and componentization, plus multi-threaded C++ retrieval communication framework smqRPC); Search Basic Services (layered retrieval services including offline indexing, online retrieval, and retrieval access); Search OSS System (offline index generation, online index rolling updates, retrieval intervention, ABTest); Effect Extension Components (decoupled scoring and ranking with built-in basic relevance ranking); API Components (SDK, smq protocol access, HTTP RESTful interfaces); and Query Analysis capabilities (word segmentation, synonyms, error correction, timeliness calculation, intent recognition, component analysis, non-breaking, new word discovery).

Engine Features

1. Query Retrieval Recall : TurboSearch supports multiple query expansions and result fusion. For example, searching "吃鸡" (chicken eating) can be expanded to "和平精英" (Game for Peace) or other hot event queries, with multiple expansion results fused to better match user intent.

2. QRW & Layered Scoring and Ranking : QRW is Tencent AI Lab's accumulated Query analysis NLP service covering correction/synonyms/non-breaking/basic relevance for vertical search, plus Query rewriting/timeliness/intent recognition/component analysis/text classification for web search. TurboSearch implements 5-layer Rank (L0-L4) covering offline, inverted intersection, fine calculation, and global fine ranking.

3. High-Performance Retrieval : Parallel retrieval uses multi-threaded parallel processing for the most time-consuming parts (intersection + L1 scoring and L3 scoring), splitting inverted indexes for parallelized retrieval. Special lock-free multi-threaded result merging design avoids CPU idle time during result merging. Weak-AND algorithm is being explored for long Query recall scenarios, combining Weak-AND with AND to balance recall rate and retrieval performance. Inverted index optimization includes BuddyAllocator with 10M/s allocation/deallocation capability, CombinedChunk and SingleSlice designs to solve storage efficiency issues.

4. Multi-Granularity Indexing : Different from N-gram brute-force indexing, multi-granularity indexing focuses on discovering implicit phrases in documents and queries, supplementing normal word segmentation. Coarse-grained word recall is performed first; if no results or too few results, fine-grained word recall is triggered.

5. Massive Data Index Support : Three index cluster types - FOB (full memory index, supports real-time add/delete), GOB/NOB (memory inverted+forward, disk summary, no real-time add/delete), WOB (disk index, no real-time add/delete).

6. Core Logic Plugin Extensions : Support for filter, score, intersect, syntax, and segment plugins.

7. Private Deployment : Supports Docker deployment for operation systems, DB environments, and service modules, with independently designed routing and resource management.

System Operations

Offline/Online operations ensure stable continuous service through smooth full data version updates without affecting online services, and seamless real-time data integration with full data rolling. The intervention system supports general intervention rules and custom intervention rules. Full-process retrieval and data diagnostics help locate recall and data issues across multi-cluster, multi-machine, multi-layer complex systems.

Application Scenarios and Outlook

TurboSearch can be applied to traditional text search, relationship chain search, LBS location-related search, and centralized cluster vector search. Future directions include continuous WAND performance optimization, knowledge entity link-based search, new structural indexes (graph, tree) for vector retrieval optimization, and gradual open-source ecosystem development.