Understanding RDMA: InfiniBand, iWARP, and RoCE Technologies and Their Differences

Remote Direct Memory Access (RDMA) is a technology that transfers data directly between the memory of two computers without involving the operating system, reducing latency and CPU load.

Originally implemented on InfiniBand networks, RDMA has been adapted to traditional Ethernet, giving rise to two main Ethernet‑based approaches: iWARP and RoCE. RoCE further splits into RoCEv1 (layer‑2 only) and RoCEv2 (layer‑3 over UDP/IP).

RDMA is increasingly important for high‑performance computing, big data analytics, AI, and IoT, where large volumes of data must be moved quickly across data‑center networks.

Traditional TCP/IP stacks suffer from high latency, multiple data copies, and interrupt handling. RDMA eliminates these overheads by allowing applications to place data directly into remote memory, bypassing the OS and TCP/IP processing.

InfiniBand (IB) implements RDMA on a dedicated fabric requiring IB NICs and switches, offering low latency, high bandwidth, and credit‑based flow control.

iWARP builds RDMA on top of TCP/IP, using the MPA, DDP, and RDMAP sub‑protocols. It works over standard Ethernet and does not require lossless Ethernet, but needs iWARP‑capable NICs and can consume more resources due to TCP connections.

RoCE transports InfiniBand‑style RDMA over Ethernet. RoCEv1 operates only within a layer‑2 broadcast domain, while RoCEv2 uses UDP/IP, enabling routing across layer‑3 networks and supporting ECN‑based congestion control and load balancing.

Both iWARP and RoCE provide a unified RDMA API, but differ in physical and link‑layer requirements: RoCE needs lossless Ethernet (PFC, ECN) and RoCE NICs, whereas iWARP relies on standard Ethernet and TCP/IP.

Choosing between these technologies involves trade‑offs in latency, throughput, CPU load, and deployment complexity; RoCE generally offers lower latency and higher throughput, while iWARP offers broader compatibility with existing Ethernet infrastructure.