High‑Performance NVMe‑oF Storage with Intel DPDK/SPDK: Architecture and Implementation Details

For many years SATA and SAS disks dominated enterprise storage; SSDs improved performance but still lag behind CPU speeds and cannot fully exploit their potential, while traditional storage suffers from long access paths, high latency, and low throughput, making it unsuitable for big‑data workloads.

NVMe, using PCIe as the SSD interface, dramatically shortens I/O paths and, with a streamlined software stack, reduces data‑access latency, representing a major breakthrough in storage technology and driving industry adoption.

Modern distributed file systems are moving to all‑flash solutions, scaling horizontally to provide block, object, and file services. Advanced hardware such as RDMA, NVMe, and NVDIMM, combined with a software‑hardware integrated design, improves reliability. To eliminate traditional bottlenecks—frequent system calls, context switches, data copies, and protocol overhead—Intel DPDK/SPDK are introduced as key user‑space foundations for NVMe‑oF, FCP, and iSCSI front‑end services.

DPDK/SPDK employ huge pages, polling, core pinning, and lock‑free mechanisms to cut CPU context‑switch and concurrency overhead, boosting I/O response speed. The SPDK BDEV framework defines upper‑layer access interfaces and lower‑layer device registration, allowing multiple protocols to share a unified block‑device abstraction and decoupling front‑end from back‑end storage.

Intel DPDK/SPDK serve as the core of the product design, integrating with a distributed architecture to fully exploit NVMe devices and RDMA channels. Early development tasks focused on feasibility analysis, validation, and prototype hardware that supports mainstream RDMA and FC adapters, implementing front‑end NVMe‑oF, FCP, and iSCSI interfaces.

SPDK is an Intel‑provided high‑performance storage development kit built on DPDK. Its user‑space, asynchronous, polling‑based application framework includes an NVMe driver and target, as well as a block‑device abstraction layer (BDEV) that enables developers to construct custom, efficient storage applications.

The NVMe‑oF concept processes all I/O stages on the same CPU core, achieving lock‑free operation and maximal performance. The transport layer abstracts underlying channels (RDMA, FC, TCP), while the BDEV layer connects to various block devices, allowing seamless integration of different storage back‑ends.

SPDK’s modular design consists of three layers: a foundation layer that initializes DPDK and establishes a core‑pinning, polling scheduler; core functional modules such as BDEV, NVMF, and NVMe that implement specific business logic; and example applications/tests that demonstrate usage and enable extensibility.

In the H3C product, the solution integrates SPDK with RDMA and FC cards to provide block storage services (NVMe‑oF, FCP, iSCSI). The architecture uses kernel‑mode RDMA drivers via the Verbs framework, while FC drivers run in user space through DPDK, unifying transport handling.

Key enhancements made during development include: adding FC transport implementation; creating a user‑space FC driver based on DPDK/UIO; refactoring interface layers for flexibility; ensuring NQN uniqueness; implementing abort command support; waiting for I/O completion on disconnect; adding proactive disconnect on the target side; introducing a keep‑alive mechanism; detailed I/O latency and IOPS statistics; validating listener legitimacy and preventing duplicate connections; adding a CLI for debugging; and fixing reliability issues such as RDMA shared‑completion‑queue errors, inaccurate connection counting, listener exception handling, and graceful process termination.

Overall, the project extends SPDK’s NVMF and iSCSI modules to meet real‑world business requirements, delivering a high‑performance, reliable, and extensible all‑flash NVMe storage solution.