Optimizing GPU Utilization for Multimedia AI Services with high_service

This article extends the previous discussion on deep‑learning advertising computation and focuses on the core problem of improving GPU utilization and avoiding GPU waste in multimedia AI service scenarios.

Challenges include low GPU usage caused by Python’s Global Interpreter Lock (GIL) in single‑process inference, high memory consumption of large models, and highly uneven traffic patterns that lead to either resource starvation during peaks or waste during low‑traffic periods.

high_service Framework is introduced as a high‑performance inference solution designed for Alibaba‑Mama’s multimedia online services. Its key components are:

Cluster‑level service scheduling with priority‑based auto‑scaling and multi‑tenant GPU sharing to handle traffic fluctuations and low‑traffic services.

Service‑architecture optimization using a CPU/GPU separated multi‑process design, which isolates the GPU inference process from CPU‑heavy preprocessing, thereby eliminating GIL‑related bottlenecks.

Model‑level acceleration via TensorRT (for PyTorch) and optimized TensorFlow pipelines, including automatic model conversion, engine caching, and removal of inference‑irrelevant operations such as spectral normalization.

The system architecture routes requests through Nginx, distributes preprocessing to multiple CPU processes, and feeds tensors to a dedicated GPU process. GPU kernels are scheduled on separate CUDA streams to maximize parallelism.

Performance results show that dynamic auto‑scaling can adjust GPU card count from a few dozen during peaks to a minimal number during off‑hours, dramatically improving overall GPU utilization.

Future Plans aim to automate bottleneck detection, further optimize CPU workloads by offloading image/video processing to GPU, and continue kernel‑level and scheduling optimizations for emerging hardware.