Real-Time AI-Powered AR Beauty Effects on the Web

Live streaming, short videos, and online meetings are increasingly using AR techniques based on AI detection and graphics rendering. While native applications have mature AR solutions, implementing AI face detection and real‑time rendering on the Web has been challenging due to performance bottlenecks.

With the continuous maturation of Web technologies, AR on the Web has become feasible. This article summarizes the key technical points for achieving AI‑driven AR beauty effects in a browser.

1. Data Acquisition – To unify input formats and support various media sources, MediaStream is used as the standard input. Video, camera, or canvas streams are converted to MediaStream for processing. In practice, large‑resolution frames are down‑sampled to ImageBitmap before feeding the detection model, which reduces texture‑decoding overhead and improves performance in frequent video scenarios.

2. Detection – Detection speed is a major bottleneck on the Web. TensorFlow.js runs at about 30 FPS due to JavaScript limitations. By leveraging WebAssembly, loading C++‑based models with SIMD optimizations, caching results from previous frames, and off‑loading computation to the GPU, the detection pipeline can approach 60 FPS.

3. Face Modeling – After obtaining facial landmarks, the points are pre‑processed and merged into a 2D mesh. To support a wider range of masks, the mesh is expanded outward using fitting algorithms, enabling full‑head coverage.

4. Spatial Positioning – Accessories such as headwear are attached to the head region based on the face model. A conversion algorithm maps the standard‑model coordinates to faces of varying size and orientation, ensuring accurate placement of stickers and masks.

5. Makeup Composition – Unlike headwear, makeup is rendered directly on the facial mesh. WebGL shaders render texture layers onto the mesh, and custom blending modes are implemented in shaders because the built‑in WebGL blend modes differ from those used in design tools like Photoshop.

Implementation Details – The positioning algorithm uses triangle coordinates: when a sticker is dragged in the authoring tool, the smallest enclosing triangle is identified, weights for the three vertices are computed, and these weights are packed into the asset protocol. The front‑end SDK then resolves the real‑time position by applying the same weights to the corresponding triangle on the detected face.

Final Effect – The solution, Tencent Cloud Vision Cube Web Beauty Effects, provides a complete pipeline (asset creation, management, front‑end integration) for Web and Mini‑Program platforms, and can be quickly combined with TRTC or live‑streaming services to enrich real‑time video experiences.