How Dragonfly Accelerates Image Distribution and Scales Kubernetes Batch Processing

From April 18‑21, 2023, CNCF held KubeCon+CloudNativeCon in Amsterdam, gathering leading open‑source communities and cloud‑native experts.

Volcano Engine’s cloud‑native team and ByteDance’s orchestration team presented two technical sessions.

Best Practices for Accelerated Image Distribution Using Dragonfly

Speakers: Wenbo Qi (Ant Financial/Dragonfly community) and Yingyang Huang (Volcano Engine/Dragonfly community).

Dragonfly is a P2P‑based image and file distribution system. The talk covered Dragonfly & Nydus architecture, design, how Dragonfly speeds model distribution in machine‑learning inference engines, best practices for image acceleration on Volcano Engine, download‑time metrics, and integration with Harbor, Nydus and other ecosystem components.

Kubernetes Batch Processing at Scale – A Scheduling Perspective

Speakers: Lim Haw Jia and Fan Deliang (ByteDance).

The session examined challenges of running massive offline workloads on native Kubernetes—diverse pod types, unified scheduling constraints, and scalability. It shared ByteDance’s experience handling nearly a million daily offline tasks, reasons and benefits of Kubernetes‑hosted batch processing, techniques such as gang scheduling and DRF, parallelizing compute‑intensive scheduling components, and the resulting improvements in resource utilization and cost.

Related open‑source project: https://github.com/kubewharf