iQIYI Network Traffic Security Monitoring (QNSM): Architecture, Design, and Applications

Large‑scale Internet companies usually have a complex, multi‑zone network boundary. Protecting and controlling such a boundary is a major challenge for security architecture.

To address this, iQIYI’s security team built the QNSM (iQIYI Network Security Monitor) engine, a full‑traffic analysis platform used in many cross‑zone security detection and control scenarios. QNSM is the core of iQIYI’s security defense system.

Network Boundary Complexity – Typical enterprise networks consist of office networks, multiple data‑center sites, CDN nodes, hybrid‑cloud connections, BYOD and wireless hotspots, all of which create fragmented “new boundaries”. These lead to fragmented, multi‑layered defense, massive traffic volumes (often >100 Gbps), and a wide range of internal and external threats.

QNSM Overview – QNSM performs full‑traffic analysis for asset discovery, monitoring, baseline modeling, anomaly detection, data extraction, ACL verification, and forensic tracing. It is built on DPDK to achieve high performance, real‑time processing, and horizontal scalability.

Key Features

High performance: leverages DPDK on standard x86 servers to process >10 Gbps per NIC, using zero‑copy, batch packet I/O, share‑nothing design, and RSS‑based CPU‑core binding.

Strong extensibility: side‑tap deployment, optical splitting, modular pipeline, configurable resources (queues, CPUs, mempools).

Multi‑dimensional features: DDoS detection data, basic DFI/DPI, Suricata‑based IDPS, IPv4/IPv6 support.

Real‑time: integrates IDPS detection with 10‑second aggregation intervals, streams enriched data to Kafka for downstream analysis.

Architecture

QNSM runs as a service on multi‑core x86 servers equipped with multiple 10 GbE/40 GbE NICs. Traffic is mirrored via SPAN or TAP, optionally split across servers. DPDK provides zero‑copy, multi‑queue packet processing.

Basic Libraries

PORT – abstraction of NIC queues and ring buffers.

MSG – lock‑free inter‑core messaging for control and data planes.

ACL – five‑tuple policy description.

TBL – wrapper for DPDK hash tables with CRUD API.

SCHED – thread scheduling, custom packet processing, timers.

Pipeline Components

SESSM – packet parsing, flow aggregation, copy‑forward.

SIP_AGG – source‑IP feature aggregation.

VIP_AGG – VIP‑plus‑sport aggregation for DDoS detection.

DUMP – PCAP capture for forensic analysis.

EDGE – exports multi‑dimensional data to Kafka.

DETECT – integrates Suricata for IDPS.

Control Layer

The Master component receives policy messages from the central control (Aegis) via Kafka and orchestrates the pipeline (e.g., enabling/disabling DUMP, SIP_AGG, VIP_AGG).

Security Applications

QNSM is used for DDoS detection, IDPS, and network DLP. For DDoS, traffic to protected VIPs is aggregated, baseline‑modeled, and anomalies are scored using interpretable models (e.g., score‑card). Detected attacks trigger automated actions: policy push to Master, activation of SIP_AGG/DUMP/VIP_AGG, and forwarding of enriched data to the security big‑data engine.

For IDPS, Suricata rules are delivered via an IDPS gateway; packets forwarded by SESSM are inspected by Detect (Suricata) and alerts are sent to Kafka for further correlation. The system also extracts protocol‑level features (HTTP, MySQL, Redis, etc.) and file metadata for DLP.

Open‑Source Status

QNSM has been deployed in more than 22 clusters (130+ analysis nodes) with a total capacity of 1 TBps. The core code is open‑sourced on GitHub (https://github.com/iqiyi/qnsm) under a standard contribution workflow.

Future Roadmap

Improve usability and configuration simplicity.

Further optimize performance and resource consumption.

Add encrypted‑traffic feature extraction.

Advance DPI/DFI capabilities.

Support NetFlow and other standard exports.

Integrate Zeek/Bro and sandbox analysis.

Enable TAP‑based bypass and firewall chaining.

Expand to broader business‑security and intelligent‑operations scenarios.

Expose cluster management, monitoring, and incident‑response functions.