Aligning the PCI‑Express Roadmap with the Cadence of Compute Engines and Networks

When system architects design the next‑generation platform, they first examine CPUs, accelerators, memory, flash, network interface cards, and PCI‑Express controllers and switches; in hybrid‑compute, memory‑centric, and shared‑accelerator clusters, switches become increasingly critical.

The problem is that these roadmaps are not truly coordinated. Most CPU and GPU vendors aim for a major compute‑engine upgrade every two years, aligning architecture and process changes with product sales. Ethernet and InfiniBand switch and NIC vendors also follow a two‑year cycle, often tying releases to Intel Xeon launches, a rhythm now disrupted by Intel’s shifting plans, AMD’s resurgence, and Arm‑based CPUs from large cloud builders.

PCI‑Express has been ubiquitous for two decades. While the PCI‑Express specification releases have generally kept pace, PCI‑Express controllers have followed the roadmap, but PCI‑Express switches from MicroChip and Broadcom have lagged far behind.

We believe all these roadmaps need better coordination. Specifically, the PCI‑SIG, which controls the PCI‑Express specification, should adopt a two‑year cadence rather than the average three‑year pace of the past twenty years. Jumping directly to PCI‑Express 7.0 on a short cadence would better serve the industry, and PCI‑Express 7.0 should be released as soon as possible to support higher I/O bandwidth and channel counts for CXL‑based layered and shared memory systems.

We have corrected the PCI‑SIG bandwidth chart, which mistakenly shows the PCI‑Express 6.0 specification released in 2022; it was actually released in 2021.

We appreciate the PCI‑SIG’s work, which has kept the PCI‑Express bus on a predictable roadmap since the late release of PCI‑Express 4.0. However, severe signal and material challenges stalled data‑center adoption of PCI‑Express 3.0 for seven years, and Intel’s early‑generation CPUs were slow to integrate newer I/O due to issues such as the burnt‑out SATA ports on the Sandy Bridge Xeon E5 chipset released in March 2012.

Typically, about a year after a PCI‑Express specification is published, controllers appear in compute engines and NICs. Thus, when PCI‑Express 4.0 debuted in 2017, the first systems using it arrived in 2018 (IBM Power9‑based servers), followed by AMD’s Rome EPYC 7002 in August 2019, and Intel finally integrated PCI‑Express 4.0 controllers into its Xeon SP processors with Ice Lake in April 2021.

Even though the PCI‑Express 5.0 specification was released in 2019, it did not become available in products until IBM’s high‑end Power E1080 with Power10 in 2021, AMD’s Genoa EPYC 9004 in November 2022, and Intel’s Sapphire Rapids Xeon SP in January 2023.

This shows that, despite a two‑year specification step, the product cadence between PCI‑Express 4.0 and 5.0 is effectively three years.

We argue that specifications and products should follow a shorter two‑year cycle so that compute engines and interconnects can be fully aligned, including PCI‑Express switch ASICs, which have traditionally lagged behind the widely adopted PCI‑Express 3.0, 4.0, and 5.0 specifications.

In every generation, the lag between PCI‑Express ports and PCI‑Express switches forces system architects to choose between composability (ideally using switches at the pod level) and raw bandwidth (providing it via direct server slots). System and cluster designs need both composability and bandwidth, and we will add high fan‑out to the mix.

Currently, only two PCI‑Express switch manufacturers exist: Broadcom (which acquired PLX Technologies years ago) and MicroChip. In February 2021 we analyzed MicroChip’s Switchtec ASIC for PCI‑Express 5.0, which expands channels from 28 to 100 and ports from 16 to 52, though it has not yet entered volume production. Broadcom introduced a PCI‑Express 5.0 chip family in February 2022, including the ExpressFabric PEX 89100 switch with 24‑144 channels and 24‑72 ports; at the time of writing we are confirming shipment status.

Our view is that PCI‑Express switches must be available from multiple vendors so that compute, memory, and storage servers can all embed chips supporting any given PCI‑Express level. Large‑scale deployment requires the ability to embed switches in servers without losing bandwidth, ports, or sacrificing fan‑out. Hence we have recently encouraged Rambus to enter the PCI‑Express switch ASIC market.

Just as the PCI‑SIG released a draft 0.3 version of the PCI‑Express 7.0 specification, all these considerations are primary factors.

Let’s look at our predictions made a year ago when the PCI‑Express 6.0 specification was completed and PCI‑Express 7.0 was on the horizon.

The PCI‑Express 7.0 specification is expected to be approved in 2025, meaning it would not appear in systems until late 2026 or early 2027. We consider this wait time too long; we need PCI‑Express 7.0 to provide bandwidth accelerators for the massive data required by AI model training and simulation, and it must match the fully complex CXL 4.0 shared‑memory and pooled‑memory specifications.

Accelerating the market introduction of PCI‑Express 7.0 controllers and switches is difficult, and all product types must speed up. Compute‑engine and peripheral vendors will hesitate to extract as much investment as possible from the PCI‑Express 6.0 product cycle.

Nevertheless, with PCI‑Express 6.0 already in products and undergoing rigorous testing (due to its new PAM‑4 signaling and FLIT low‑latency encoding), the industry should begin accelerating and align as closely as possible with CPU and GPU roadmaps, maintaining a two‑year cadence.

Keeping all components balanced and moving forward together is essential.

Source: Semiconductor Industry Observation Original article: https://www.nextplatform.com/2023/07/07/pci-express-must-match-the-cadence-of-compute-engines-and-networks/