AI‑Driven Audio Content Understanding and Safety for Live Streams

Guest: Qi Lu, Senior AI Expert at Shumei Technology. Editor: Jin Weiyun. Source: AI Science Frontier Conference. Community: DataFun. Note: Please credit the source when reposting.

Why use machines to understand audio content? When massive amounts of audio data exist, manual comprehension becomes difficult, whereas images and text can be processed more efficiently; therefore, machine assistance is essential for scalable audio analysis.

What aspects of audio must a machine understand? Primarily the contextual scenario of the content.

1. Content Safety: In 2018 a popular livestream host was banned for inappropriate remarks; similar bans have occurred for political speech. Foreign hostile groups also broadcast propaganda audio/video on platforms. Additionally, livestreams contain pornographic audio/video and advertising diversion. Detecting and labeling these behaviors in audio helps platforms manage such risks.

2. Content Operation: Understanding audio enables recommendation, e.g., recognizing a male voice to suggest a younger female user, or a female voice to suggest a younger male, thereby extending interaction time.

Core Idea: Transform unstructured audio/video into structured data by assigning labels, which facilitates downstream tasks such as interception and recommendation.

3. Solution: The proposed workflow consists of four steps:

• Audio Segmentation: Split long recordings into short segments.

• Speech‑to‑Text: Convert speech to textual transcription.

• Recognition: Tag both the text and the audio.

• Synthesis: Aggregate segment results to produce a final sentence or video.

Challenges include far‑field recognition, reverberation, and noise. Live hosts often add reverb to singing, which degrades speech recognition. Casual speaking and singing also introduce pitch variations that standard ASR models struggle with.

Audio Segmentation (VAD): A mainstream method that uses deep neural networks (DNN) to predict silent versus non‑silent frames, followed by windowing to obtain segment boundaries.

Speech‑to‑Text (ASR): Utilizes a DNN+LSTM acoustic model trained with lattice‑free MMI and an n‑gram language model to extract textual content from audio.

Beyond transcription, tasks such as music detection, pornographic audio detection, etc., are addressed with a sound‑classification framework. Data augmentation is applied to balance rare sound classes, and a TDNN+bi‑GRU+Attention architecture is employed.

Text Recognition: Classifies transcribed text (e.g., pornographic or abusive topics) using fastText or traditional machine‑learning models, combined with keyword matching after preprocessing steps such as tokenization and normalization.

Behavior Recognition: Detects coordinated malicious activities (e.g., repeated playback of pre‑recorded propaganda) using logistic‑regression scoring on device/IP features, flagging suspicious behavior for further review.

The overall architecture integrates ASR, text‑related, sound‑related, behavior‑related models, and a whitelist database. Model scores are fed to a rule engine that aggregates results and produces final decisions.

Real‑world impact: On a live‑stream platform, user reports catch ~1 violating audio per day, manual random checks catch ~20, while Shumei’s AI system catches ~160 violations daily, also providing timestamps and transcriptions.

Author Introduction: Qi Lu, senior AI expert at Shumei Technology, holds a master’s degree from Nankai University, with ten years of frontline AI experience at Baidu and 360, now leads voice & text products at Shumei.

About Shumei Technology: Founded in June 2015, Shumei is a leading AI anti‑fraud solution provider recognized as a high‑tech enterprise, offering real‑time, end‑to‑end fraud and content‑security solutions across finance, e‑commerce, video, live‑stream, audio, social media, travel, education, real estate, and more, serving thousands of enterprises worldwide.

—END—

Machine Learning in Business Applications?

Join the DataFun community: reply with "DF" to the official account.

DataFun is a practical data‑intelligence community that hosts offline deep‑tech salons and online content curation, aiming to disseminate industry experts' practical experience to data scientists and AI practitioners.