Hot Upgrade of OVS‑DPDK in Didi Cloud: Achieving Millisecond‑Level Downtime

Based on the open‑source OVS‑DPDK project, the Didi Cloud engineering team introduced innovative improvements that enable millisecond‑scale hot upgrades while preserving the high‑performance forwarding capability of the original solution.

Background

Didi Cloud initially adopted an OpenStack‑style approach, using kernel‑mode OVS to build an SDN overlay network. During the practice, several pain points were identified:

Insufficient performance

Inability to perform hot upgrades

High development difficulty

High maintenance cost

Prototype Design

To address these issues, the team evaluated existing solutions and collaborated closely with Intel. They selected OVS‑DPDK as the foundation and built a data‑flow model as shown in the diagram below.

Key Data‑Layer Transformations

1. Hardware‑related (VF) modifications

Using VF‑based flow classification, overlay traffic is directed to a virtual function (VF) while other host traffic continues to be processed by the kernel via the physical function (PF). OVS‑DPDK only handles the VF, leaving PF traffic untouched, which simplifies OVS‑DPDK processing and isolates its stability from non‑SDN traffic.

2. Middle‑layer: OVS packet processing

The solution provides a stateless forwarding path for VM traffic. The VM traffic is split into two parts: outer overlay header handling and inner packet forwarding, both realized through OVS‑DPDK flow tables. Because conntrack is disabled, the forwarding remains stateless, which greatly benefits hot‑upgrade reliability.

3. Upper‑layer: vhost‑user and VM interaction

QEMU is used as the vhost‑user server. The OVS‑DPDK process connects to QEMU via a Unix socket. By modifying QEMU to support two redundant connections, hot upgrades can switch between the old and new OVS‑DPDK processes without service interruption.

Memory model adopts 2M/4K pages to minimize impact on existing VMs and prepare for future migrations.

Solution Advantages

1. Short upgrade time

Typical hot‑upgrade solutions in the industry take seconds. In Didi’s framework, each VM experiences an upgrade window of about 80 ms, making the network interruption virtually invisible to users.

2. Good scalability

The upgrade time is independent of the number of VMs. By switching VMs one‑by‑one from the old OVS‑DPDK process to the new one, even hundreds of VMs can be upgraded within a few seconds while each VM’s interruption remains ~80 ms.

3. Fast fault recovery

Two independent OVS‑DPDK processes run in parallel; a standby process is pre‑started and fully initialized (including vf2). If the primary process crashes, the standby takes over instantly, achieving recovery times comparable to the hot‑upgrade duration.

Test Data

Performance: Single‑core throughput ≈ 4 Mpps

Hot upgrade: VM network interruption ≈ 80 ms

DPDK version: 17.11

OVS version: 2.9.0

QEMU version: 2.9

CPU: Intel(R) Xeon(R) CPU E5‑2630 v4 @ 2.20 GHz (kernel 3.10.0‑514.16.1.el7.x86_64)

NIC: Ethernet Controller X710 for 10GbE SFP+ 1572