WeChat NLP Algorithm Microservice Governance: Challenges and Solutions

The talk introduces the topic of WeChat NLP algorithm microservice governance, highlighting the difficulties of handling a large number of algorithm microservices and the solutions developed to address them.

Overview : Using the example of WeChat Reading recommendation, the speaker explains how even a small feature involves numerous microservice calls for feature retrieval, recall, and ranking, illustrating the massive scale of microservice interactions in the app.

Management challenges include efficient development, deployment, and operation of many algorithm microservices. Solutions presented are automated CI/CD pipelines that generate microservice scaffolding from a Python function, task‑aware auto‑scaling, and a DAG/DSL framework for visualizing, testing, and deploying composed services.

Performance challenges focus on optimizing inference for deep‑learning models. The team adopts specialized inference frameworks, kernel optimizations, and a custom Python interpreter called PyInter that runs Python scripts with near‑C++ performance while sharing GPU memory across threads, avoiding GIL contention.

Scheduling challenges involve dynamic load balancing across many identical microservices. The speaker describes the limitations of static benchmarking, real‑time status polling, and centralized queues, and introduces an improved algorithm called Joint‑Idle‑Queue (JIQ) that builds on the Power‑of‑2‑Choices method with idle‑queue and amnesia components to reduce tail latency.

The summary reiterates the three main challenges—management, performance, and scheduling—and the corresponding solutions: automation of development pipelines, model‑aware performance optimizations (including PyInter), and the JIQ load‑balancing algorithm that significantly compresses P99/P50 latency.