Deep Dive into Intel DPDK/SPDK and NVMe‑oF Architecture and Implementation

For many years SATA and SAS drives dominated enterprise storage; although SSDs improved performance over AHCI, they still lag behind CPU speeds and cannot fully meet the demands of big‑data storage due to long access paths, high latency, and low throughput.

NVMe, using PCIe as the SSD interface, dramatically shortens I/O paths and, with a streamlined software stack, further reduces data‑access latency, making NVMe storage the industry’s emerging trend.

Modern distributed file systems are moving toward all‑flash solutions, scaling horizontally to provide block, object, and file services. Advanced hardware such as RDMA, NVMe, and NVDIMM, together with user‑space implementations based on Intel DPDK/SPDK, address traditional bottlenecks like frequent system calls, context switches, data copies, and protocol‑stack overhead.

DPDK/SPDK introduce huge pages, polling, core pinning, and lock‑free mechanisms, cutting CPU overhead and improving I/O response. SPDK’s BDEV framework abstracts storage devices, offering a unified interface for multiple protocols (NVMe‑oF, FCP, iSCSI) and allowing independent testing and faster development.

Intel DPDK/SPDK serve as the core of the product design, integrating with distributed architecture to exploit full‑flash NVMe devices and RDMA channels, delivering high‑performance features. The current prototype supports major RDMA and FC adapters and implements front‑end NVMe‑oF, FCP, and iSCSI interfaces.

SPDK NVMe‑oF Overview – SPDK builds on DPDK, providing a user‑space, asynchronous, polling‑based programming framework, a high‑performance NVMe driver, and a block‑device abstraction layer (BDEV) that enables custom storage applications.

The SPDK application framework manages CPU cores and threads, offers efficient inter‑thread communication, and implements a lock‑free I/O processing model.

These mechanisms together create a low‑latency, high‑IOPS solution where the BDEV layer connects storage protocols to various back‑end devices, and the NVMe‑oF target runs in user space, naturally suited for NVMe‑over‑Fabrics.

NVMe‑oF’s core idea is to keep all I/O stages on the same CPU core, eliminating locks and maximizing performance. The transport layer abstracts underlying channels (RDMA, FC, TCP), while BDEV abstracts block devices, enabling flexible target implementations.

SPDK’s modular design consists of three layers: a base environment that initializes DPDK and sets up a polling scheduler; core functional modules such as bdev, nvmf, and nvme; and example applications/tests that demonstrate extensibility.

SPDK NVMe‑oF Application Solution – The H3C product integrates SPDK with RDMA and FC cards to provide block storage services (NVMe‑oF, FCP, iSCSI). The solution uses Intel’s Verbs framework for RDMA in kernel mode and DPDK’s user‑space mechanisms for FC.

Key enhancements made during development include:

Implementation of FC transport in the nvmf module.

User‑space FC driver based on DPDK and UIO.

Re‑architected interface layering for better flexibility and reduced coupling.

Unique NQN generation combining subsystem NQN with node identifiers.

Support for abort commands.

Graceful I/O completion handling during link teardown.

Active link termination on the target side.

Target‑side keep‑alive mechanism to detect network failures.

Detailed I/O latency and IOPS statistics.

Legitimacy checks for listeners and prevention of duplicate connections from the same host/port.

CLI tools for dynamic query, creation, and configuration.

Reliability fixes such as handling shared RDMA completion‑queue errors, accurate link‑count checks, listener exception handling, and safe process termination.

Overall, the project extends SPDK’s nvmf and iSCSI modules to better suit real‑world business scenarios, delivering a high‑performance, user‑space storage stack.