Implementing Face Blocking Danmaku Using Machine Learning in Browser

This article discusses the implementation of face-blocking danmaku (bullet comments) using machine learning in the browser. The author, Liu Jun, a senior development engineer at Bilibili, explains the traditional methods of implementing face-blocking danmaku and introduces a new approach that leverages machine learning models to extract human contours in real-time from video frames.

The traditional methods include pre-processing videos to extract human regions and storing them as SVG files, or using SEI (Supplemental Enhancement Information) to embed SVG data into video streams for live broadcasts. The new approach involves using machine learning models like MediaPipe SelfieSegmentation to extract human contours in real-time and applying them as a mask to the danmaku layer.

The article details the optimization process, including model selection, performance tuning, and the use of OffscreenCanvas and Web Workers to offload CPU-intensive tasks. The final implementation achieves a CPU usage of around 5% on a 2020 M1 MacBook, making it suitable for production use.

The author also discusses the challenges faced during the implementation, such as performance bottlenecks and compatibility issues, and provides insights into potential applications of the technology, such as background replacement, face blurring, and virtual accessories.

The article concludes with a summary of the optimization process and a list of related articles for further reading.