WebRTC Testing and Quality Assurance for Live Streaming

Author : Wang Haojie, Senior Test Development Engineer at Bilibili.

Background : Real‑time communication (RTC) refers to real‑time audio/video transmission, commonly implemented with WebRTC. WebRTC, initiated by Google, provides audio/video capture, encoding/decoding, data transport, and rendering, enabling peer‑to‑peer connections across platforms (Web, native C++). Bilibili’s live‑streaming services (video linking, PK, voice rooms, etc.) rely on WebRTC.

Quality Assurance Plan : High quality is required for RTC services; any jitter, delay, audio‑video sync issues, or noise degrade user experience. The testing team provides dedicated quality assurance through specialized testing of the WebRTC SDK, signaling, and media services.

Test Objects

Client SDK

The client SDK includes:

• Audio‑video processing module – capture, encode/decode, playback/render.

• Network transmission module – packet encapsulation, transport, decapsulation.

• Signaling transmission module – connection establishment, signaling send/receive.

RTC Server

Server side consists of signaling server and media server.

• Signaling server – coordinates and manages audio/video transport, bridges business and media services.

• Media server – forwards, mixes, and distributes media streams. When acting as an SFU (Selective Forwarding Unit), it forwards streams between clients, often deployed on CDN edge servers for better quality.

Test Design

Functional Testing

Two perspectives guide test case design:

1. Module‑level testing – verifies each capability (audio/video processing, network, signaling) independently, including performance and weak‑network resilience.

2. Regression testing – validates the entire WebRTC‑based business flow after SDK updates.

Media Processing Module Test

Focuses on audio/video capture, encoding/decoding, rendering, and device‑specific compatibility (Android, iOS, Windows, different chipsets). Compatibility and performance are key concerns.

Signaling Module Test

Validates signaling sequences for actions such as join/leave, mute/unmute, kick, PK, etc., and checks for lost, duplicated, or out‑of‑order messages.

Network Transmission Module Test

Involves testing various protocols (TCP/UDP, ICE/STUN/TURN, RTMP/RTSP/RTP, HTTP/HTTPS, WebSocket) and requires fine‑grained network simulation.

Weak‑Network Testing

Key metrics:

Delay – >150 ms for audio, >400 ms for video becomes noticeable.

Packet loss – high loss degrades video continuity; combined with delay it can cause severe quality drop.

Bandwidth – insufficient bandwidth leads to frame drops, audio glitches, and overall degradation.

Jitter – variance in packet arrival time; high jitter can cause buffer under‑run and audio/video artifacts.

Network Test Methods

Custom network rules can be applied via packet‑loss emulators or professional network‑damage tools. Example for macOS using pfctl and dnctl :

在Mac OS X系统中，你可以使用pfctl和dnctl命令来设置UDP和TCP的丢包率
创建一个名为pf.conf的文件
dummynet in proto udp all pipe
1
dummynet out proto udp all pipe
1
dummynet in proto tcp all pipe
2
dummynet out proto tcp all pipe
2
创建一个名为pipe.sh的脚本
#!/bin/sh
dnctl pipe
1
config plr
0.2
dnctl pipe
2
config plr
0.1
运行以下命令来启动pfctl
sudo pfctl -f pf.conf
运行pipe.sh脚本来设置网络条件
sh pipe.sh
以上的设置将设置UDP的丢包率为
20
%，TCP的丢包率为
10
%。可以根据需要调整这些参数。

On Windows, the tool clumsy is used for similar simulations (lag, drop, throttle, duplicate, out‑of‑order, etc.).

Client Performance

Memory leaks can occur during video rendering or UI teardown after a link ends. Performance testing with tools like perfdog monitors CPU, memory, GPU, frame rate, temperature, and power consumption. Example: a recent iteration increased GPU usage from 23.1 % to 45.5 % during PK due to a frame‑rate parameter change.

Server Performance

WebRTC rooms/channels organize users for real‑time communication. RPC requests to the signaling server and long‑link broadcast services must be functionally verified and load‑tested. Media servers, acting as SFUs, require stress testing with multiple roles sending/receiving streams to assess load.

Online Quality Observation

Two main metrics:

• QoS (Quality of Service) – objective data such as packet counts, RTT, bitrate, loss, jitter, rendering delay.

• QoE (Quality of Experience) – user‑centric data like join success rate, first‑frame render time, decode success, stutter rate, end‑to‑end latency.

The Bilibili RTC team collects end‑to‑end data from SDKs and servers to quickly locate issues. When QoE degrades, the team cross‑references user feedback with QoS logs to pinpoint root causes.

References

[1] WebRTC Official Documentation (Chinese) – https://github.com/RTC-Developer/WebRTC-Documentation-in-Chinese

[2] WebRTC API – https://developer.mozilla.org/zh-CN/docs/Web/API/WebRTC_API

[3] Weak‑Network Testing – https://zhuanlan.zhihu.com/p/349827871

[4] Jitter vs. Latency – https://webrtc.org.cn/jitter-latency/

[5] WebRTC Weak‑Network Simulation – https://webrtc.org.cn/network-test-for-webrtc/