Intelligent Live‑Streaming Video Editing Techniques and Practices

Live streaming has become a major e‑commerce channel, but raw streams are long and contain many irrelevant sections. Alibaba Mama proposes an end‑to‑end intelligent clipping system that automatically extracts high‑quality short videos for ad delivery.

The solution consists of several modules: (1) Segment Decomposition – split the live stream into 3‑10 minute clips based on product highlights; (2) Speech Classification – transcribe audio via ASR, then classify the text into 21 semantic tags (e.g., product style, feature, benefit) using a GPT‑based multi‑label model; (3) Visual Selection – group clips with similar speech tags and rank them by visual appeal; (4) Storyline Arrangement – combine speech and visual categories into diverse narrative sequences; (5) Effect Processing – add cover images, BGM, background replacement, subtitles, transitions, stickers, etc.

Key technical details include:

3.1 Speech Classification – a multi‑label classifier trained on a large, manually annotated corpus from various product categories, achieving 96 % accuracy.

3.2 Highlight Detection (PLD‑VHD) – a pixel‑level distinction model that learns viewer attention patterns to assign a highlight score to each visual segment.

3.4 Speech Coherence Scoring – a sentence‑pair model (based on GPT embeddings) that predicts a coherence score (0‑1) for adjacent clips, improving the smoothness of stitched videos.

3.5 Direct‑to‑Highlight Tags – fine‑grained tags derived from content analysis that allow users to jump to specific sections (e.g., “benefit introduction”).

3.6 Video Material Selection – multiple script variants are generated per industry (e.g., beauty), tested in online ad experiments, and the best‑performing version is selected for deployment.

The system has been validated in real‑time ad pipelines, boosting creative quality and advertising efficiency. Future work will explore deeper content understanding and broader application scenarios.