Multimedia Content Understanding at Weibo: Video Summarization, Quality Assessment, OCR, Embedding, and CV‑CUDA Optimization

Weibo processes large volumes of multimedia content—videos, images, audio, and text—to support various downstream services such as recommendation, moderation, and copyright protection, and this article first outlines the overall architecture of its content‑understanding system.

For video summarization, both static (single‑frame cover) and dynamic (short clip) approaches are discussed, including classic methods like dppLSTM (ECCV 2016), SUM‑GAN (CVPR 2017), CSNet and DR‑DSN (AAAI 2018), and Weibo's own weakly supervised model that selects representative and diverse segments without costly manual labeling.

Video quality assessment is tackled by extracting frame‑level features with a CNN, modeling temporal relations with a GRU, and aggregating scores; Weibo further refines this with a hierarchical Transformer that produces frame embeddings, segment embeddings, and a final video‑level quality score, achieving better alignment with its specific business scenarios.

Text recognition (OCR) evolves from two‑stage detection‑then‑recognition pipelines to end‑to‑end models such as FOTS; Weibo adapts this architecture for Chinese text by separating detection and recognition feature towers, substantially improving accuracy on diverse image sources.

Video embedding is designed at three granularities—frame, segment, and whole‑video vectors—trained via contrastive learning on both visual and audio streams; these embeddings enable use cases like fine‑grained copyright checks, efficient deduplication, tag reuse, and large‑scale clustering for recommendation.

To accelerate the inference pipeline, Weibo replaces the traditional CPU‑based decode‑preprocess‑GPU model flow with CV‑CUDA, performing JPEG decoding and preprocessing directly on the GPU, which reduces data transfer overhead, balances CPU/GPU utilization, and yields up to 70% higher throughput.

The article concludes with a Q&A session addressing preprocessing bottlenecks, resource utilization, training frameworks, batch processing details, and practical deployment considerations.