CPU Share Syncer: Enabling High‑Priority Task CPU Preemption in iQIYI Video Production Kubernetes Clusters

This article presents iQIYI's practice of optimizing video production workloads on Kubernetes (K8s) clusters by allowing high‑priority tasks to obtain more CPU resources and finish faster.

Background : Video production jobs have varying business priorities and CPU requirements (e.g., ultra‑HD tasks need 8 CPUs, normal tasks need 4 CPUs). Native K8s does not differentiate business priority, so high‑ and low‑priority tasks with the same CPU request receive identical allocations. The goal is to let high‑priority jobs pre‑empt CPU from low‑priority ones.

Principle Overview : A daemon process (cpu‑share‑syncer) runs on each physical node. It reads the priority annotation of each Pod and raises the cpu.shares value in the pod’s Cgroup, giving high‑priority pods a larger CPU weight and allowing them to pre‑empt lower‑priority pods. This requires disabling the K8s CPU CFS quota (via --cpu-cfs-quota=false ) so that pods can freely use idle CPU.

High‑Priority CPU Preemption : Pods are scheduled with CPU Resource Requests and CPU Resource Limits . By turning off the CFS quota, the limit no longer caps CPU usage. Users add an annotation iqiyi.com/cpu-share (integer, unit = 1/1024 of a CPU) to indicate the desired weight. For example, a pod that should be able to use 10 CPUs sets the annotation to 10240 . The daemon writes this value into the pod’s cpu.shares file.

Overall Design : cpu‑share‑syncer is deployed as a DaemonSet, mounting the host’s Cgroup path via a hostPath volume. It communicates with the kube‑apiserver using a List‑Watch to obtain pod information, extracts the iqiyi.com/cpu-share annotation, and updates the corresponding cpu.shares file. Pods without the annotation keep their default weight (e.g., 1024 for 1 CPU).

Usage : Deploy the daemon as a DaemonSet on every node. When creating a high‑priority pod, add the annotation with the desired weight. Existing pods can be patched to add or modify the annotation, and the daemon will adjust the weight periodically (default sync period ≈ 10 s, matching kubelet’s CPUManager).

Technical Details : The daemon runs continuously, periodically reconciling the annotation‑specified weight to the pod’s cpu.shares . This periodic update prevents loss of weight after pod restarts. The sync interval is configurable. The daemon’s List‑Watch approach reduces load on the kube‑apiserver compared to full list queries.

Test Results :

1. Single‑node functional test : Two pods with identical CPU Resource Requests (10) but different iqiyi.com/cpu-share values (8192 vs. 16384) ran stress --cpu 40 . Prometheus metrics showed CPU time ratio ≈ 2:1, confirming weight effect.

2. Multi‑pod pre‑emption test : Three baseline pods (10 CPUs, no weight) and one high‑weight pod (weight 262144) ran the same stress command. The high‑weight pod’s CPU time grew rapidly while the earlier pods’ CPU time remained flat, demonstrating pre‑emption.

3. Production gray‑scale test : In a real K8s cluster, 50 % of video‑production tasks were given weight 262144, the other 50 % left unchanged. The weighted tasks achieved ~16 % lower execution‑time‑to‑media‑size ratio, confirming efficiency gains.

Conclusion and Outlook : By disabling CPU CFS quota and using cpu‑share‑syncer, iQIYI enables high‑priority video‑production tasks to obtain preferential CPU access, improving execution speed. The daemon currently runs only on worker nodes; future work includes integrating priority awareness into the scheduler to distribute high‑priority pods more evenly across nodes.