Challenges and Techniques in Distributed Training of Large Language Models

Introduction – Since 2019, large language models have advanced quickly, bringing both exciting research results and significant infrastructure challenges for practitioners.

Historical Background – The rapid growth of LLMs is likened to a gold‑rush, requiring more sophisticated tools (e.g., “excavators”) and supporting infrastructure such as transport links and fuel reserves.

Distributed Training Challenges

1. Massive compute demand – Compute roughly equals six times the product of model parameters and training tokens; GPT‑3, LLM‑65B, and similar models require 10^23–10^24 FLOP‑scale resources, costing tens of millions of RMB per training run.

2. Model size vs. GPU memory – Parameter growth doubles roughly every 3.9 months, pushing memory needs to terabyte levels; single‑GPU solutions become infeasible, and even high‑end GPUs like A100 would need centuries to finish training alone.

3. Building distributed systems – Large clusters (e.g., 2048 GPUs) can reduce training time to days, but require careful handling of network topology, CPU‑GPU coordination, and multi‑level communication.

Distributed Training Technology System

1. Fundamental concepts – Early work (AlexNet, ImageNet) laid the groundwork; later, frameworks such as TensorFlow, All‑Reduce, Word2Vec, and DDP evolved to support large‑scale training.

2. Data parallelism – DeepSpeed’s ZeRO optimizations (Stage 1‑3) reduce optimizer state, gradient, and activation memory by partitioning them across devices; off‑load techniques move data to host memory or disk at the cost of extra compute time.

3. Pipeline parallelism – Synchronous pipelines split the model into stages, passing activations between devices; techniques like 1F1B scheduling, micro‑batching, and round‑robin stage partitioning improve device utilization and reduce idle time.

4. Tensor parallelism and other optimizations – Strategies such as matrix slicing, GR=2 style partitioning, and fine‑grained task scheduling further boost efficiency in multi‑GPU environments.

Future Challenges

• Development of intuitive, feature‑rich visualization and scheduling tools for large‑scale training.

• Research on automatic parallelism search, dynamic batch sizing, and heterogeneous hardware coordination.

• Exploration of new model architectures beyond Transformers (e.g., WW‑KV) and their impact on distributed frameworks.

Q&A

Q1: Optimizing beyond Transformers? – Discusses the need for specialized methods when scaling to trillion‑parameter models.

Q2: Efficient memory use in smaller settings? – Mentions recompute and trade‑offs between memory and compute.

Q3: Machin vs. DeepSpeed adoption? – Highlights their different focuses (3D parallelism vs. ZeRO+off‑load) and complementary nature.

Q4: Automatic parallel strategy search? – Emphasizes the importance of IR representation and hardware‑aware scheduling.

Q5: Parallelism of Softmax? – Describes the two‑step communication (All‑Reduce Max and All‑Reduce Sum) required for distributed Softmax.

Conclusion – The session summarized the challenges, current techniques, and open research problems in large‑model distributed training.