Understanding RDMA (Remote Direct Memory Access): Background, Related Work, and Technical Details

Remote Direct Memory Access (RDMA) is a technology designed to eliminate OS involvement in data transfer, enabling high‑throughput, low‑latency communication by moving data directly between the memories of two computers.

Background

Traditional TCP/IP communication requires data to be copied from user space to kernel space, encapsulated through multiple protocol layers, and then copied back to user space on the receiver side, incurring significant latency and CPU overhead.

Network performance is measured by bandwidth and latency; modern workloads often involve small messages where processing overhead dominates.

TCP/IP suffers from I/O bottlenecks and high context‑switch costs, limiting achievable bandwidth on high‑speed networks.

Related Work

High‑performance networking has evolved through technologies such as TCP Offloading Engine (TOE), User‑Net Networking (U‑Net), Virtual Interface Architecture (VIA), and finally RDMA.

TOE moves TCP/IP processing to the NIC; U‑Net moves the entire protocol stack to user space, removing kernel copies; VIA standardizes a user‑level network interface.

RDMA Overview

RDMA offers three key properties: low latency, low CPU overhead, and high bandwidth.

It defines two types of verbs:

Memory verbs: RDMA read, write, and atomic operations that bypass the remote CPU.

Messaging verbs: RDMA send and receive, which involve the remote CPU.

RDMA can operate in reliable or unreliable modes and supports both connected (RC) and datagram (UD) transports.

Hardware Implementations

Three major RDMA hardware families exist: InfiniBand, iWARP (Internet Wide Area RDMA Protocol), and RoCE (RDMA over Converged Ethernet). InfiniBand provides native reliability; RoCEv1 runs over Ethernet with lossless flow control, while RoCEv2 uses UDP over Ethernet.

RDMA Architecture

RDMA places a specialized RNIC (RDMA‑enabled NIC) between the application and the network, exposing a verbs interface that allows user‑space programs to directly read/write remote memory without kernel intervention.

RDMA Operation Flow

Application issues an RDMA read or write request; the request is sent from user space directly to the local NIC.

The NIC reads the local buffer and transmits the request over the network.

The request carries the remote virtual address, memory key, and payload.

The remote NIC validates the key and writes/reads data directly to/from the remote buffer.

Completion notifications are delivered via Completion Queues (CQ).

Single‑Sided Operations (READ/WRITE)

These operations require only the initiator to know the remote address and key; the remote side does not participate, making them ideal for bulk data transfer.

Double‑Sided Operations (SEND/RECEIVE)

Both endpoints must post matching work requests; SEND/RECEIVE are typically used for control messages, while bulk data moves via READ/WRITE.

Conclusion

RDMA fundamentally changes network communication by bypassing the kernel, reducing copy overhead, and delivering low‑latency, high‑throughput transfers, which are essential for modern high‑performance computing and data‑center workloads.