Coupang’s Distributed Cache Architecture Accelerates AI/ML Model Training

Coupang, a Fortune 200 technology company, originally used a multi‑cluster GPU architecture for AI/ML model training but faced four major challenges: lengthy data preparation and copy times, low GPU utilization, rising storage costs, and heavy operational burden from localized data silos.

To address these issues, the AI platform team introduced a distributed‑cache system with five key innovations: automatic data ingestion from a central data lake, dramatically faster data loading, a unified data‑access path for model developers, automated data‑lifecycle management, and seamless scalability to Kubernetes environments.

The new architecture delivers six concrete benefits: faster model‑training speed, reduced storage costs, higher cross‑cluster GPU utilization, lower operational overhead, improved portability of training jobs, and roughly a 40% I/O performance gain over traditional parallel file systems.

Technically, the solution combines an AWS multi‑region cloud deployment with on‑premise GPU clusters in a hybrid model. A distributed cache layer runs on NVMe‑equipped instances (cloud) or NVMe‑disk CPU nodes (on‑prem), caching only hot data to cut storage costs and eliminate manual cleanup.

Cache access is provided via a FUSE pod that presents a POSIX‑compatible filesystem to training containers; the pod forwards I/O requests to backend worker pods that read/write data from local NVMe pools or fetch missing pages from the data lake. An etcd service maintains mount tables and worker membership, ensuring consistent data paths across clusters.

For model developers, the architecture means instant data availability, no code changes needed for cross‑region data access, higher GPU utilization by allowing flexible task scheduling, and faster training due to reduced I/O latency. For platform engineers, it lowers storage and operational costs, simplifies scaling and maintenance through Kubernetes operators, and provides tools for pre‑warming caches.

In summary, Coupang’s distributed‑cache‑based AI platform accelerates model training, improves resource efficiency, cuts costs, and enhances developer experience while supporting seamless, multi‑region GPU resource orchestration.