How Alibaba’s MediaAI Studio Brings AI‑Powered Live Stream Interactions to Life

Introduction

Hello everyone, I am Pan Jia from Alibaba Taobao Multimedia Front‑end, also known as Lin Wan. I am honored to share at the 15th D2 conference.

How to greet in the live stream?

During the Chinese New Year, we wanted hosts to be able to wish fans in the live room and add festive effects. The design lets the host perform a greeting gesture, which triggers real‑time rendering of holiday decorations, and facial recognition adds props such as a lucky‑money‑god hat.

Creating gesture greeting effects

The effect is built in four steps: (1) Designers create static or animated assets (e.g., a lucky‑god hat) using design software; (2) Assets are assembled in our self‑developed MediaAI Studio editor, where frame adaptation, face‑following, gesture triggers, and local preview are configured; (3) The asset package is uploaded to the content platform; (4) Hosts select the package in the streaming client, and the effects are rendered and merged into the stream in real time.

Examples include triggering flower‑text, couplets, or fireworks by making a heart or greeting gesture, and adding a lucky‑god hat that follows the host’s forehead.

Media‑Intelligent Solution Design

Traditional “red‑packet rain” overlays a separate H5 page on the video stream, which is disconnected from the content. Our media‑intelligent approach renders assets directly inside the video stream, allowing hosts to control the rain via gestures, thereby increasing interaction rate and viewer dwell time.

The solution combines AI/AR gameplay with the video stream, aiming for a rapid production cycle of “7+3+1” days: 7 days for algorithm development, 3 days for gameplay scripting, and 1 day for asset creation.

The end‑to‑end chain consists of four stages: asset production, asset management, asset usage, and asset display. Producers use the editor to create gameplay, the ALive platform manages assets, hosts enable the gameplay in the streaming client, and the live container renders the effects using SEI key‑frames.

Smart assets are defined by a JSON protocol that describes modules such as filters, stickers, beauty effects, and text templates. The rendering engine downloads the assets, parses the configuration, and performs real‑time compositing.

Interactive gameplay examples include the Double‑11 “Super Cup” challenge where the host moves a cup with body gestures, and the “Pop Mart” challenge where facial tracking controls a character.

Technical flow: MediaAI Studio generates asset packages and scripts; ALive creates a component that binds the gameplay; the host enables the component via a control panel; the streaming client downloads and executes the script, merging assets into the stream; the player extracts SEI key‑frames to locate interactive hotspots.

MediaAI Studio Editor

Built on Electron, MediaAI Studio is a desktop editor powered by the cross‑platform rendering engine RACE, which integrates the MNN inference framework and PixelAI algorithms. The main process handles window management, while the renderer process provides the UI, real‑time preview, and a worker thread that communicates with the RACE native module.

The RACE C module is exposed to JavaScript via a Node.js native addon, enabling JS scripts to control rendering, canvas updates, and module configuration through JSON and binary protocols.

From a designer’s perspective, the editor supports creating smart stickers, face‑tracking props, and gesture‑triggered effects.

From a developer’s perspective, the editor allows scripting gameplay logic, such as controlling a bird’s trajectory with face detection.

Future Plans

Media‑intelligent tools are still early; we aim to integrate deeper with the platform, including algorithm, asset, and publishing services. The editor will support secure front‑end production, project creation, debugging, code review, and deployment. We also plan to open the ecosystem to designers, ISVs, and commercial partners to scale interactive live‑stream experiences.

Live‑Stream Q&A

Q1: What front‑end work is involved in effect development (aside from the asset platform)?

A1: The workflow includes production, management, usage, and display. Front‑end builds the MediaAI Studio editor (Electron), integrates with ALive for management, provides PC and app streaming tools, and drives the interactive components in the live room.

Q2: How is effect detection frequency chosen?

A2: Detection runs only when a gameplay is enabled; each algorithm has its own frame‑rate settings, separating detection frames from follow‑up frames to reduce overhead.

Q3: Where are recognition and merging performed, and what protocols are used?

A3: Both are executed on the host’s streaming client (PC or app) using standard live‑stream protocols such as RTMP for pushing and HLS/HTTP‑FLV for playback.

Q4: Does merging increase latency, and how is interaction latency ensured?

A4: Merging does not add latency; slow algorithms may lower frame rate. User‑side interactions are handled locally; for tight sync scenarios we use SEI+CDN to align video and data.

Q5: Recommended open‑source library for gesture detection?

A5: Google’s MediaPipe – https://github.com/google/mediapipe

Q6: Does recognition significantly increase front‑end bundle size?

A6: No, the bundle mainly contains assets and scripts; the heavy models run on the device side.

Q7: What framework powers the editor’s algorithms, TensorFlow.js?

A7: Not TensorFlow.js; we use the MNN inference engine and PixelAI platform, integrated via the RACE rendering framework.

Q8: Are red‑packet positions random, and how are hot‑zones defined?

A8: Positions are random; the streaming script encodes location, size, and transformation into SEI frames, which the player parses to reconstruct interactive hot‑zones.

Q9: How is game code performance ensured?

A9: Game logic runs in C++ via a Node.js native addon, offering near‑native speed. Future plans include exposing a WebGL interface to leverage mainstream H5 game engines for richer interaction.