Performance Comparison of Kubernetes Container Network Solutions: Flannel, IPvlan, VXLAN, Host‑GW, and AWS‑VPC

Kubernetes requires each container in a cluster to have a unique routable IP address, but it does not allocate IPs itself, delegating this task to third‑party networking solutions.

The goal of this research is to identify the solution with the lowest latency, highest throughput, and lowest installation cost for latency‑sensitive workloads, focusing on performance at 30‑50 % of maximum load to represent typical non‑overloaded usage.

Competitors

Docker with --net=host – This reference setup lets containers inherit the host’s IP address, avoiding any network containerization and serving as a baseline because no network container can outperform it.

Flannel – A widely‑tested, production‑ready virtual network maintained by the CoreOS project. When a node with Flannel joins the cluster, it (1) uses etcd to allocate a subnet, (2) creates a virtual bridge (docker0), and (3) configures a packet‑forwarding backend such as AWS‑VPC, host‑gw, or VXLAN.

AWS‑VPC – Registers the node’s subnet in an Amazon VPC table, limited to 50 entries, thus supporting at most 50 machines and only works on AWS.

Host‑gw – Creates IP routes to subnets via the remote machine’s IP; requires direct layer‑2 connectivity between hosts running Flannel.

VXLAN – Creates a virtual VXLAN interface; because Flannel forwards packets through a bridge, each packet traverses two network stacks, adding overhead.

IPvlan – A Linux kernel driver that creates virtual interfaces with unique IPs without a bridge, simplifying packet forwarding and promising better performance, though it lacks mature automation tools, raising installation cost.

Load‑test methodology

For each candidate we:

Connect two physical machines.

Run tcpkali in a container on one machine to generate a constant request rate.

Run nginx in a container on the other machine to serve fixed‑size files (350 B or 4 KB).

Capture system metrics and tcpkali results.

Benchmarks were executed at request rates from 50 k to 500 k RPS. Responses of 350 B (100 B payload, 250 B header) and 4 KB were used.

Key results

IPvlan consistently showed the lowest latency and highest throughput. Host‑gw with Flannel followed closely, performing better than VXLAN in all tests. At the highest loads (450 k RPS, 350 B) IPvlan’s latency was about 30 % higher than --net=host but still outperformed other solutions. Host‑gw outperformed AWS‑VPC, whose 50‑machine limit and AWS dependency were drawbacks.

For 4 KB responses the ranking was similar, though the maximum sustainable RPS was lower (≈270 k RPS for a 10 Gbps NIC). Detailed latency percentiles for various setups at 50 k, 150 k, and 250 k RPS are presented in the original tables.

Test environment

Two Amazon c4.8xlarge instances (CentOS 7) with enhanced networking, NUMA‑aware 36‑core CPUs, 10 Gbps NICs, and 60 GB RAM were used. Linux kernel 4.3.0 with the ixgbevf driver enabled IPvlan support.

Network tuning included:

IRQ affinity: first core of each NUMA node handles NIC interrupts.

RPS configuration via echo "00000000,0003fdfe" > /sys/class/net/eth0/queues/rx-0/rps_cpus (and rx‑1).

XPS configuration via echo "00000000,07fc01ff" > /sys/class/net/eth0/queues/tx-0/xps_cpus (and tx‑1).

RFS flow entries set to 65 536.

nginx workers (18) pinned to specific CPUs using worker_cpu_affinity .

tcpkali run under taskset -ac 0-17 with scheduler tuning ( echo 10000000 > /proc/sys/kernel/sched_migration_cost_ns ).

Additional sysctl tweaks (e.g., fs.file-max=1024000 , net.ipv4.tcp_low_latency=1 ) and Red Hat tuned profile for network latency were applied, and NOTRACK rules were added to reduce nf_conntrack overhead.

Based on the results, the authors selected host‑gw with Flannel as the primary solution due to its low dependency footprint and adequate performance, while keeping IPvlan as a backup for future use when Flannel adds IPvlan support.