How Terway Implements Elastic ENI and Advanced CNI Designs for Kubernetes

1. CNI Concept

Container Network Interface (CNI) is invoked by the Container Runtime Interface (CRI). The interface is provided as an executable binary that receives JSON‑encoded input via stdin and returns a JSON‑encoded result via stdout, completing a CNI call. See the official spec for plugin delegation.

2. Common Implementation Approaches

Binary and script plugins are maintained in the containernetworking/plugins repository.

CNCF‑provided CNI implementations use various Linux kernel features to create container network interfaces supporting modes such as bridge, ipvlan, vlan, etc. Built‑in IPAM plugins include dhcp, static, and host‑local.

Plugins Supplied

Main: interface‑creating

bridge : Creates a bridge and adds the host and container to it.

ipvlan : Adds an ipvlan interface in the container.

loopback : Sets the loopback interface state to up.

macvlan : Creates a new MAC address and forwards traffic to the container.

ptp : Creates a veth pair.

vlan : Allocates a VLAN device.

host-device : Moves an existing device into a container.

dummy : Creates a dummy device in the container.

Windows: Windows specific

win-bridge : Creates a bridge and adds the host and container.

win-overlay : Creates an overlay interface for the container.

IPAM: IP address allocation

dhcp : Runs a daemon on the host to make DHCP requests on behalf of the container.

host-local : Maintains a local database of allocated IPs.

static : Allocates a single static IPv4/IPv6 address, useful for debugging.

Daemon socket

When business logic becomes complex, a daemon process (e.g., Cilium, Terway) listens on a domain socket. CNI triggers are redirected to the daemon, which performs the core handling, solving concurrency and logging limitations of pure binary plugins.

Terway Overview

1. Elastic ENI Support and Component Design

Elastic Network Interface (ENI) can hold multiple secondary IPs (6‑20 per ENI). Terway assigns these secondary IPs to containers, greatly increasing pod density. Terway supports Vethpair policy routing and ipvlan modes.

Vethpair Mode

Ipvlan Mode

Linux kernels 4.2+ support ipvlan virtual networking, allowing multiple sub‑interfaces with distinct IPs on a single physical NIC. Terway binds ENI secondary IPs to ipvlan sub‑interfaces, simplifying network topology and improving performance.

Terway ENI‑Trunking Architecture

Trunk ENI is a virtual NIC bound to a VPC ECS instance, offering higher instance density, independent security groups, and VSwitch capabilities.

Resource Management and Allocation

Terway manages network resources via Alibaba Cloud APIs. It records pod‑resource associations and lease periods, enabling IP retention for StatefulSets and garbage‑collection to prevent leaks.

Pod Resource Management

When a pod is deleted, its IP is retained until lease expiration, then returned to the pool, ensuring stable IPs for updates and avoiding resource leaks.

Resource Pooling

Resources are cached; when the pool falls below a low watermark, new resources are provisioned, and excess resources are released when above a high watermark, ensuring efficient utilization.

Scheduling

Different instance types have varying ENI quotas (e.g., ecs.g6.xlarge supports two secondary ENIs). Terway reports ENI capacity via the Kubernetes DevicePlugin, allowing pods to request aliyun/eni: 1 and enabling the scheduler to respect node limits.

Network Policy

In Veth‑based VPC and ENI‑multi‑IP policy‑routing modes, Terway integrates Calico’s Felix component for NetworkPolicy enforcement. In ipvlan mode, it integrates Cilium for the same purpose.

Pod Traffic Control

Terway reads ingress and egress annotations on pods and applies tc token‑bucket‑filter (tbf) rules on the pod’s network interface to limit bandwidth.

ENI‑CNI Design

Network Layer Design

Using trunk ENI, each trunk creates ipvlan sub‑interfaces (sub_ports) that can bind multiple IPs, providing independent security groups and VSwitch capabilities while leveraging existing OVS implementations.

CNI Layer Design

ENI‑CNI Design Idea

Terway’s UltronServer ENI‑trunk API operates on sub_ports. It binds a pod’s ipvlan sub‑interface to an unused secondary IP on a sub_port, handling IP allocation (IPAM) directly.

Differences from Original VPC‑CNI

The new design simplifies IPAM logic, removing frequent web requests to UltronServer for port, fixed IP, and floating IP operations. IPAM responsibilities shift from the controller to the agent, eliminating a single point of overload and failure. The agent now handles all CNI‑related operations per node.

Each node binds a single trunk ENI; if traffic exceeds capacity, additional ENIs can be provisioned. IP pool management is delegated entirely to Ultron, with the CNI only synchronizing state.

Future Work

Current NetworkPolicy enforcement relies on OVS; future work may involve direct NIC integration. Observability improvements (metrics, tracing, logging) and exploring eBPF for performance and network optimization are planned.