Understanding DMA and RDMA: Principles, Advantages, and Protocols

What is DMA

DMA (Direct Memory Access) allows peripheral devices to read from or write to memory without CPU involvement. Without DMA, an I/O device such as a network card must request the CPU to copy data between memory and the device, consuming CPU cycles that could be used for computation.

With a DMA controller on the bus, the controller handles the entire data copy: it moves data from memory to its internal registers and then to the I/O device, while the CPU only initiates and finishes the transfer.

What is RDMA

RDMA (Remote Direct Memory Access) enables a node to directly read from or write to the memory of a remote node, bypassing the TCP/IP stack. The remote side is unaware of the transfer, and most of the work is performed by hardware rather than software.

In traditional networking, data moves from the user space to kernel buffers, passes through the TCP/IP stack, and is copied by the CPU at both ends. With RDMA, the local NIC DMA‑copies data directly from user memory to the remote NIC, which then places it into the remote user memory, eliminating most CPU involvement.

Advantages of RDMA

RDMA is widely used in high‑performance computing (HPC) and large data centers, though the hardware is more expensive than ordinary Ethernet NICs.

Zero‑copy : Data is not copied between user space and kernel space, reducing latency and CPU load.

Kernel bypass : Data can be prepared in user space and handed directly to the hardware, avoiding system calls and context switches.

CPU offload : The remote CPU does not participate in the data transfer; packet framing and parsing are handled by the NIC hardware, freeing CPU cycles for other work.

These features give RDMA higher bandwidth and lower latency compared with traditional Ethernet, making it ideal for bandwidth‑sensitive and latency‑sensitive workloads.

Protocols

RDMA technology is implemented by three main protocols: Infiniband (IB), RDMA over Converged Ethernet (RoCE), and iWARP.

Infiniband

IB, defined by the InfiniBand Trade Association (IBTA) in 2000, provides a complete link‑to‑transport stack but requires dedicated IB switches and NICs.

RoCE

RoCE leverages Ethernet at the link layer. RoCE v1 uses the IB transport, while RoCE v2 encapsulates IB traffic in UDP/IPv4, allowing routing over standard Ethernet networks.

iWARP

iWARP is an IETF‑standardized protocol built on TCP. Because TCP incurs higher overhead and memory usage, iWARP is less popular than RoCE.

All three protocols require specialized NIC hardware.

Standard and Ecosystem Organizations

The IBTA (founded 1999) maintains the Infiniband specifications and runs compliance testing. The OpenFabrics Alliance (OFA, founded 2004) develops the open‑source OFED software stack that supports IB, RoCE, and iWARP.

Community and Software

The Linux kernel includes an active RDMA subsystem. User‑space APIs such as libibverbs provide socket‑like interfaces for all three protocols. The main user‑space repositories are rdma-core (core libraries, drivers, documentation) and perftest (performance testing tools).

UCX is a communication framework built on top of RDMA and other high‑performance transports, offering a higher‑level API for HPC applications.

Hardware Vendors

Key vendors of IB/RDMA hardware include Mellanox (now part of NVIDIA), Huawei, Intel (via QLogic acquisition), Broadcom, Marvell, and Fujitsu. Mellanox’s latest Connext‑X‑6 series supports 200 Gb/s, while Huawei’s Kunpeng 920 chip supports 100 Gb/s RoCE.

Users

Major companies such as Microsoft, IBM, Alibaba, and JD.com have deployed RDMA in data centers and HPC clusters. Adoption is growing as the performance benefits become more compelling.