Understanding NVMe over Fabrics: Architecture, Transport Options, and Use Cases

NVMe over Fabrics (NVMe‑oF) is a specification released by the NVM Express working group in 2016 that extends the low‑latency, high‑throughput characteristics of NVMe beyond a single host by connecting servers to storage over a network.

Similar to how Fibre Channel (FC) SAN extended SCSI, NVMe‑oF moves NVMe from a local PCIe link to a fabric, enabling the protocol’s benefits to scale from individual servers to entire data‑center environments.

By replacing PCIe with fabrics such as RDMA or Fibre Channel, NVMe‑oF greatly improves flexibility and scalability, allowing NVMe SSD resources to be accessed remotely while preserving low latency and high concurrency.

The main transport options are:

NVMe/RDMA (NVMe‑RDMA) : Offers the highest performance but requires lossless Ethernet, DCB, PFC, and other complex network features; suited for HPC, distributed databases, and AI workloads.

FC‑NVMe (NVMe over Fibre Channel) : Ideal for environments that already have FC SAN infrastructure; uses 16 Gb or 32 Gb FC HBAs and can coexist with SCSI over FC.

NVMe/TCP : Approved in November 2018; runs over standard Ethernet without special hardware, making deployment easy though with lower performance than RDMA or FC‑NVMe.

NVMe‑oF emerged because storage performance has outpaced local PCIe connections; modern SSDs and persistent memory deliver orders‑of‑magnitude lower latency, demanding faster networks to share these resources across many hosts.

Disaggregating compute and storage (the “storage‑compute separation” trend) enables pooling, shared usage, improved fault isolation, cost reduction, higher utilization, and easier management, aligning with Software‑Defined Storage principles.

Typical deployment scenarios include JBOF (Just a Bunch of Flash) and EBOF (Enterprise Bunch of Flash) systems, where dozens of SSDs are aggregated to provide up to 16 M IOPS, far beyond what a single server can consume.

Future directions point toward fully distributed all‑flash architectures based on NVMe‑oF, the rise of DPUs, and scalable hyper‑converged infrastructures where storage can be shared across nodes without each node needing local disks.