Common Video Experience Metrics and Laboratory Testing Methods for Real-Time Communication

In a previous DevforDev column, our engineers shared the establishment and practice of Agora's reference‑free video quality assessment (VQA) system. Building a VQA system takes a long process, but many methods exist to evaluate real‑time video quality, which we summarize here.

We generally assess video quality using metrics such as latency, stutter rate, rendering frame rate, subjective visual quality, first‑frame time, and audio‑video sync. The following sections explain each metric and describe laboratory testing methods.

Latency

Video latency measures the end‑to‑end delay from capture to rendering, expressed in milliseconds; lower values indicate better real‑time performance. The largest latency contributors are video capture/encoding (part1) and client playback (part4), while transmission (part2, part3) adds minimal delay.

Laboratory testing method: Use a millisecond‑accurate timer on a PC, stream video from a camera/phone to a server, and simultaneously play it on another device. Capture both source and playback frames in a single screenshot and calculate the time difference. The procedure mirrors industry practice.

Stutter Rate and Frame Rate

Stutter rate reflects video smoothness: total stutter duration divided by total test duration, expressed as a percentage. Frame rate is the rendering frame rate observed at the receiver; higher values indicate smoother playback.

Laboratory testing method: Connect two phones, stream a rotating globe and a presenter, impose weak network conditions (e.g., 500 KB uplink, 20 % packet loss), record the stream for three minutes, then use a script to calculate stutter periods (>200 ms) and frame rate.

Subjective Visual Quality

Subjective quality evaluates how the human eye perceives video after capture, encoding, transmission, and decoding. It can be assessed subjectively (human scoring) or objectively (mathematical models). The following rating scale ranges from 10 (clear, no artifacts) to 1 (severely blurred, unrecognizable).

Laboratory testing method: Observe the video and assign a quality score based on the described criteria.

First‑Frame Time

First‑frame time measures the delay from entering a channel to displaying the first video frame, in milliseconds; a value under 1 second is desirable.

Laboratory testing method: Capture screenshots at regular intervals after playback starts; the first frame whose similarity to a reference exceeds 90 % is considered the first frame. The time difference between channel entry and this frame gives the first‑frame latency.

Audio‑Video Sync

Audio‑video sync indicates the alignment between video frames and audio; acceptable range is –200 ms to +200 ms. The closer to 0 ms, the better the sync.

Laboratory testing method: Record the playback on a phone while a computer continuously plays a test video with a visible timecode. Capture both video and audio, then compare the lip‑movement with the audio to compute the delay, using frame numbers or a sliding marker (33 ms per frame at 30 fps).

Bandwidth Tracking

Bandwidth tracking monitors the bitrate transmitted within a channel.

Laboratory testing method: Use Wireshark with a virtual RVI interface (created via rvictl -s ) to capture network packets, filter for source (client) and destination (server) IPs, and visualize I/O graphs to obtain real‑time bandwidth usage.

These metrics and their corresponding laboratory testing methods can be referenced for daily development in RTC scenarios.