GPU Cluster Scaling: Understanding Scale‑Up and Scale‑Out for AI Pods

This article, originally titled “GPU Cluster: What Are Scale‑Up and Scale‑Out?”, introduces the concept of an AI Pod—a pre‑configured, modular infrastructure that bundles compute, storage, networking, and software to accelerate AI workloads.

AI Pods combine these components into a tightly integrated unit, leading to the terms “vertical scaling” (Scale‑Up) and “horizontal scaling” (Scale‑Out). A rack in a data center can be viewed as an AI Pod; Scale‑Up adds more resources (CPU, memory, storage) to a single pod, while Scale‑Out adds more pods and connects them together.

The article defines XPU as a generic term for any processing unit (CPU, GPU, NPU, TPU, DPU, FPGA, ASIC) and lists common examples. Each XPU blade typically contains 2‑8 XPU devices, which may be single‑chip or multi‑chip modules.

AI workloads require massive data movement, both within a pod (intra‑pod) and between pods (inter‑pod). Intra‑pod communication occurs among servers in the same rack, demanding ultra‑low latency and high bandwidth, while inter‑pod communication spans racks or physical infrastructure.

Advantages of Horizontal Scaling (Scale‑Out)

Provides long‑term scalability by adding pods as demand grows.

Allows easy down‑scaling by removing pods when load decreases.

Enables use of commodity servers; no need for large monolithic machines.

Disadvantages of Horizontal Scaling

May require application re‑architecture for distributed operation.

Increases network complexity and demands robust service discovery.

Data consistency across pods can be challenging for AI training workloads.

The article compares networking standards for inter‑pod communication, noting that InfiniBand has long been the low‑latency, high‑bandwidth choice, while Super Ethernet (UEC) is emerging as an open, Ethernet‑compatible alternative.

Vertical Scaling (Scale‑Up) adds more CPU, memory, and storage to a single pod or server. An example shows a pod growing from 1 CPU/2 GB RAM/100 GB storage to 4 CPU/8 GB RAM/500 GB storage, enabling higher request throughput.

Advantages of Vertical Scaling

Simpler implementation for monolithic or single‑node applications.

Leverages powerful server hardware with modern CPUs, AI accelerators, NVMe storage, and fast networking.

Disadvantages of Vertical Scaling

Limited by physical hardware constraints; eventually hits CPU, memory, or storage ceilings.

Resource bottlenecks can appear (e.g., memory saturation while CPU remains under‑utilized).

High‑end servers are costly, making large‑scale vertical scaling expensive.

For intra‑pod communication, NVLink has been the NVIDIA standard, while the vendor‑agnostic UALink is positioned as the future high‑speed inter‑XPU link.

Source: Semiconductor Industry Observation.