Knowledge Base Retrieval Matching: Algorithm and Engineering Service Practice

This article introduces the algorithm and engineering service practice of knowledge base retrieval matching, which is a fundamental NLP problem for semantic similarity computation in enterprise knowledge management systems.

1. Background : Knowledge base retrieval matching involves given a query and numerous candidate documents, finding the Top-K most relevant documents. This technology applies to search engines, intelligent customer service, knowledge retrieval, and information recommendation.

2. Architecture : The system consists of overall architecture and request链路 (request chain) design for handling retrieval requests.

3. Algorithm Implementation :

• DSL Rewriting with PageRank : Using PageRank algorithm to calculate importance scores (PR values) for documents based on citation counts. The formula: new score = old score * log(1+2*PageRank)

• Text Recall : Traditional keyword-based retrieval using ES inverted index and tf-idf similarity calculation. Limitations include ignoring semantic meaning and inability to handle polysemy and synonyms.

• Vector Recall : Uses a dual-tower model with embedding layer, two transformer layers, and MLP. The model encodes query and document titles into vectors, then computes cosine similarity in Milvus vector database. Offline: document vectors are computed and stored in Milvus. Online: query vector is computed and compared with stored vectors.

• Fine Ranking : Uses in-batch negative sampling with InfoNCE loss for contrastive learning. Training data is generated from user click behavior. The model learns to pull similar samples closer and push dissimilar samples apart in the semantic representation space.

• Ranking Optimization : Converts ranking task to vector recall by computing mean similarity between query vector and all document title/similar question vectors. This approach solves RT (response time) and QPS issues while maintaining performance.

4. Engineering Implementation :

• Offline Training (DP Platform) : Corpus preprocessing, vectorization using the model, and batch import to Milvus cluster.

• Online Inference (Sunfish Platform) : One-click training, task management, model version management, and deployment.

• Milvus Vector Search : Supports various indexes (FLAT, IVF_FLAT, IVFSQ8H, IVFSQ8) and uses Inner Product (IP) for distance calculation. After normalization, IP is equivalent to cosine similarity.

• Algorithm Interface Services : ai-service handles model inference and Milvus retrieval; ai-app handles business logic.

5. Service Practice : Applied to official website help center and similar product recommendation.

6. Summary : Future directions include: 1) Using graph neural networks to improve embedding learning for low-frequency or new documents; 2) Modularizing word embedding models; 3) Ensuring performance and stability under high accuracy and high QPS.