Understanding Video SDK Architecture, Playback Principles, and Testing Practices

With increasing network speeds and decreasing bandwidth costs, multimedia formats such as audio and video are becoming the primary means of information transmission, making a stable and extensible player core essential.

Why not use the system's playback capability? Built‑in system players (e.g., Android's SurfaceView with MediaPlayer) support limited container formats and have poorer decoding performance, which cannot meet complex business scenarios.

Advantages of a video SDK include cross‑platform support (Android, iOS, PC), high reusability, support for almost all common container formats, and a clear separation between SDK and business logic, providing strong extensibility.

Player principle introduction

A basic video playback process consists of three parts: video decoding, video rendering, and audio‑video synchronization.

Most developers choose ffmpeg as the video development library because its code structure is clearer and its documentation more comprehensive than alternatives like GStreamer or VLC. After compiling a custom ffmpeg (e.g., version 4.0), six shared libraries and header files are generated; these are often packaged into a single dynamic library for easier crash analysis.

Video decoding

Decoding transforms streams such as H264/H265 into pixel data (e.g., YUV420, RGB565). Before decoding, a container file (e.g., video.flv) must be demuxed to extract raw video frames and audio streams. ffmpeg creates three AVFormatContext structures for file input, video output, and audio output, copies stream information, and writes separate files.

Decoding can be performed in software or hardware mode; hardware decoding reduces CPU load but may cause compatibility issues, so a fallback to software decoding is necessary.

Video rendering

Android provides several views for image display, such as SurfaceView, TextureView, and GLSurfaceView. SurfaceView consumes less memory, while GLSurfaceView enables GPU rendering via OpenGL ES. For Windows, libraries like libSDL can be used for cross‑platform rendering.

Audio‑video synchronization

Three schemes exist: (1) using an external clock for both audio and video, (2) using video timestamps to adjust audio, and (3) using audio DTS as the reference for video playback. The third scheme is preferred because video frame jitter is less perceptible to users.

Player testing

The testing phase revealed several pain points: (1) SDK version control in continuous integration, (2) frequent black‑screen and playback‑failure reports, and (3) first‑frame loading latency and stutter analysis.

1. SDK CI version control

The SDK is packaged as an AAR and uploaded to a private Maven repository. Snapshot versions are used for development, while release versions are used for testing and production. Gradle cache issues caused missing native .so files; the solution was to auto‑increment snapshot version numbers and manually manage release versions.

2. Black‑screen and playback‑failure diagnosis

After SDK refactoring, hardware decoding caused ~20 daily black‑screen reports on devices from Oppo, Vivo, Xiaomi, etc. By collecting video metadata (resolution, bitrate, codec) from user reports, reproducing the issue on similar devices, and fixing a transcoding bug, the overall black‑screen rate dropped by 60%.

3. First‑frame loading latency and stutter analysis

Log analysis moved from single‑machine scripts to Spark Streaming for real‑time processing, with cleaned data stored in HBase, then transferred to MySQL and Elasticsearch for fast queries. Optimizations reduced first‑frame load time by ~65% and stutter rate by 0.5% through BYTERANGE support and CDN coverage improvements.

Conclusion

For C‑end products, focusing on user experience, data‑driven optimization, and systematic testing of video SDKs can significantly improve playback stability and performance.