Inside Cerebras Architecture: How Co‑Designed Hardware Accelerates Deep Learning

Massive ML Demand Challenge

Neural‑network models have grown exponentially, increasing compute requirements by over three orders of magnitude in just a few years. Traditional training and inference systems cannot keep pace, so improvements of at least one order of magnitude are needed across multiple system components.

Core Architecture

Cerebras designs a tiny core (38 000 µm²) whose half area is a 48 KB SRAM and the other half a 110 k standard‑cell logic block, running at 1.1 GHz with a peak power of 30 mW. Each core embeds its own local memory, eliminating the bandwidth gap between DRAM and the compute datapath.

The 48 KB SRAM is organized as eight 32‑bit single‑port banks, providing more raw bandwidth than the compute datapath needs. Each core can perform two full 64‑bit reads and one 64‑bit write per cycle, and every core’s memory is independently addressable—there is no shared memory in the traditional sense. A 256‑byte software‑managed cache sits close to the datapath for ultra‑low‑power access to frequently used structures.

Full‑Performance BLAS Levels

With this memory bandwidth the cores can execute matrix operations at every BLAS level, not just GEMM. The architecture can sustain full‑performance AXPY (vector‑scalar) operations, which is essential for accelerating unstructured sparse workloads because sparse GEMM decomposes into many AXPY kernels.

Tensor‑Instruction Support

The core implements a fully programmable ISA with arithmetic, logic, load/store, compare, and branch instructions, all operating on locally stored data. On top of this, dedicated tensor instructions execute on a 64‑bit datapath backed by four FP16 FMAC units. The ISA treats tensors as first‑class operands, enabling direct 3‑D and 2‑D tensor FMAC operations.

Each core holds 44 Data‑Structure Registers (DSRs) that carry a pointer to a tensor plus its shape, length, and size, allowing the hardware to stream 4‑D tensors directly from memory without software‑managed tiling.

Fine‑Grained Data‑Flow Scheduling

Computation is data‑triggered: the Fabric delivers a data element, the core looks up the corresponding instruction, and execution proceeds. Zero‑valued weights are filtered out before reaching the core, so only non‑zero data triggers work, achieving up to ten‑fold sparse‑utilization over GPUs.

The core also hosts eight micro‑threads—independent tensor contexts that can be switched each cycle. A priority scheduler monitors tensor I/O availability and issues partial‑sum (PSUM) or final‑sum (FSUM) commands to coordinate reductions.

Scale‑Up: Amplifying Moore’s Law

Cerebras builds a wafer‑scale engine (WSE‑2) that is 56 × larger than the biggest CPU, covering 46 000 mm², containing 2.6 trillion transistors and 850 000 cores. The CS‑2 system integrates this chip into a standard data‑center rack, delivering cluster‑level performance in a single enclosure.

The Fabric uses a 2‑D mesh topology with 5‑port routers (four 32‑bit bidirectional links plus a core port), providing single‑cycle hop latency and low‑overhead flow control. Data packets consist of a 16‑bit FP16 payload plus 16‑bit control, forming a 32‑bit ultra‑fine‑grain packet. Static routing colors (up to 24 per link) enable non‑blocking, time‑multiplexed communication across the entire wafer.

Extending the mesh across scribe lines (<1 mm) uses advanced metal layers for high‑speed inter‑die links. Redundant static‑routing and error‑correction state machines guarantee uniform Fabric behavior even with wafer‑level defects, delivering roughly an order‑of‑magnitude bandwidth per area and two orders of magnitude better energy efficiency than GPU‑equivalent area.

Weight Streaming for Ultra‑Large Models

Model weights reside in an external MemoryX device and are streamed layer‑by‑layer into the CS‑2 system. Each weight triggers an AXPY operation; after use the weight is discarded, eliminating on‑chip weight storage limits. Gradients flow back to MemoryX during back‑propagation.

For transformer models, the 850 000 cores act as a single massive matrix‑multiply engine. Batch and sequence dimensions are mapped onto the mesh’s y‑axis, while the hidden dimension maps onto the x‑axis, enabling efficient weight broadcast and reduction.

Scale‑Out: Why It Is Hard Today

Current cluster scaling relies on data parallelism (simple but memory‑intensive) or model parallelism (complex, with quadratic activation memory growth). Neither approach scales cleanly to the largest models because compute and memory become tightly coupled, leading to intricate distributed‑system constraints.

Historical GPU‑cluster training data show that as models grow, more parallelism types are required, increasing system complexity and development effort while often yielding sub‑optimal scaling.

Cerebras Architecture Simplifies Scaling

Because a single wafer‑scale chip can host the entire model, scaling reduces to pure data‑parallel replication. The SwarmX interconnect sits between MemoryX and multiple CS‑2 systems, broadcasting weights and aggregating gradients using a tree topology that scales linearly with the number of systems.

Conclusion

ML workloads have grown by more than three orders of magnitude and will continue to do so. Cerebras meets this challenge by delivering order‑of‑magnitude improvements in core architecture (through unstructured sparse acceleration), wafer‑scale scale‑up, and truly scalable cluster scale‑out. The result is a single‑chip system capable of 75 PFLOPS FP16 sparse performance (or 7.5 PFLOPS dense) that can run the largest models without partitioning, making massive neural networks accessible to a shrinking set of organizations.