Baidu Baige’s Breakthrough: Orchestrating Giant LLM Inference with Silent Instances

1. Problem Overview – The Impossible Triangle

Deploying LLM inference at the scale of hundreds of billions or trillions of parameters faces three coupled constraints: model scale, cost/elasticity, and efficiency/stability. Traditional cloud‑native stacks cannot simultaneously satisfy these, leading to bottlenecks in large‑model deployment.

2. Automated Orchestration

2.1 FedDeployment – Atomic Instance Abstraction

Ba​idu Baige introduces a custom Kubernetes CRD called FedDeployment . The controller creates a FedReplicaSet which in turn creates FedInstance objects. A FedInstance aggregates all Pods that belong to a single logical inference instance (which may be spread across many physical nodes) into an atomic unit that can be created, updated, or deleted with a single command.

Unified lifecycle management : Users operate only on a FedDeployment resource; the controller automatically creates/updates the underlying FedReplicaSet and FedInstance hierarchy.

Replica consistency and scaling : FedReplicaSet guarantees the desired number of healthy FedInstances. Scaling is performed atomically without manual Pod handling.

Native canary releases : By defining two FedReplicaSets (e.g., v1 and v2‑canary) traffic can be shifted gradually, enabling safe version roll‑out.

2.2 Gang Scheduling – All‑or‑Nothing Coordination

Distributed inference requires every Pod in a FedInstance to be ready before processing can start. Baidu Baige implements a lightweight Init‑Barrier Container that runs in each Pod, writes its status (IP, rank, etc.) to a shared ConfigMap, and opens the barrier only when all Pods have reported ready. The reconciler then injects the full member list as an environment variable into all Pods, enabling NCCL‑based communication without additional orchestration steps.

Status synchronization and waiting: each Pod runs an Init‑Barrier, updates the shared ConfigMap, and the barrier opens only when all Pods are ready.

Service discovery information collection: Pods publish their IP and rank to the ConfigMap.

Network topology injection: the FedInstance reconciler reads the collected information, assembles a member list, and injects it into the Pods’ environment for NCCL communication.

2.3 SplitService – Unified View for Prefill/Decode Separation

LLM inference is split into a compute‑intensive Prefill stage and a latency‑sensitive Decode stage. SplitService creates separate ReplicaSets for Prefill and Decode, then presents them as a single logical service (Single Service View). This design provides:

Proportional co‑scaling: Prefill and Decode instances are scaled together according to a predefined ratio.

Network‑aware placement: Paired Prefill/Decode Pods are preferentially placed on the same node or rack to minimize KV‑Cache transfer latency.

Bin‑packing: Different resource requirements of Prefill and Decode Pods are packed efficiently to reduce fragmentation.

Dynamic load‑aware adjustment: Real‑time monitoring of Prefill load triggers automatic addition of Prefill Pods when a bottleneck is detected.

3. Adaptive Elastic Scaling

3.1 Adaptive HPA – Predictive & Simulation‑Based Decision Loop

The native Kubernetes HPA reacts only to CPU/Memory metrics, which is insufficient for LLM workloads. Adaptive HPA adds three subsystems:

Multi‑dimensional input & intelligent decision : Combines Prophet‑based traffic forecasts, real‑time TTFT and token‑per‑second metrics, operational plans, and SLO constraints.

Planning & simulation : A fast simulator (using performance baselines and dynamic programming) evaluates different Prefill/Decode ratios and instance counts, producing a safe, gradual scaling path.

Efficient execution : The Adaptive HPA controller issues scaling commands, can put instances into a silent (sleep) state or wake them instantly, achieving second‑level response.

3.2 Silent Instances – Second‑Level Activation

Cold‑starting a trillion‑parameter model can take up to ten minutes. Silent Instances keep the GPU process paused, off‑load model weights and KV‑Cache from HBM to DRAM, and release GPU compute resources. When traffic rises, weights are re‑loaded from DRAM to HBM and the instance becomes active in <30 s . When traffic falls, the instance returns to silent mode in <10 s , freeing GPU resources while preserving context.

Activation : <30 s to reload weights and resume processing.

Deactivation : <10 s to off‑load weights and release GPU.

4. High‑Performance Traffic Scheduling

4.1 Staggered Batched Scheduler (SBS) – “Bus” Scheduling

Traditional FCFS dispatch causes in‑engine queuing because each inference engine batches internally. SBS solves this with two steps:

Batching : Aggregate requests arriving within a tiny time window into a batch.

Staggered dispatch : Predict which instance will finish next and assign the whole batch to that instance just before it becomes idle, eliminating internal waiting and dramatically reducing TTFT.

4.2 DP Balancing – Eliminating Compute Bubbles

In data‑parallel (DP) execution, uneven request loads create idle periods (“bubbles”). SBS leverages the batching window to obtain global request information (prompt length, token count) and runs a greedy algorithm that distributes the batch across DP units so that each unit receives a comparable compute load.

Global information : Estimate per‑request compute cost using prompt length and token count.

Greedy load balancing : Assign requests to DP units to keep total load balanced, removing bubbles and improving overall GPU utilization.

5. Architecture Summary & Quantitative Benefits

The system is organized into four tightly coupled layers:

Foundation Layer : FedInstance abstracts a distributed set of Pods into an atomic inference unit.

Orchestration Layer : SplitService provides a unified service view for Prefill and Decode, handling co‑scaling, placement, and dynamic ratio adjustment.

Performance & Efficiency Layer : SBS eliminates in‑engine queuing and DP bubbles; Silent Instances deliver second‑level elasticity.

Intelligence Layer : Adaptive HPA predicts traffic, simulates scaling outcomes, and executes optimal scaling decisions automatically.

Key outcomes :

Stable, scalable deployment of trillion‑parameter models via FedInstance and SplitService.

Silent Instances reduce cold‑start time from >10 minutes to <30 seconds, dramatically improving resource utilization.

SBS cuts TTFT by 30‑40 % and increases system throughput by 15‑20 %.

Adaptive HPA provides full‑time, predictive autoscaling, lowering operational cost.

Overall, Baidu Baige delivers a cost‑effective, high‑performance, fully automated AI compute foundation for next‑generation LLM services.