Paddle.js: Baidu's Browser-Based AI Inference Engine for Frontend Development

Background

As Baidu's AI strategy deepens, Baidu AI capabilities are being actively deployed across various application domains. To enable AI capabilities in browser-based web frontend scenarios, Baidu's technical teams developed Paddle.js, a framework that helps frontend engineers more easily integrate intelligent features into web pages, enabling web frontends to achieve broader capabilities.

Why Introduce AI Capabilities in Frontend

Browser-based web applications offer lower barriers to access and wider reach. Integrating intelligent features can significantly enhance user experience. Traditionally, intelligent features were implemented on the server due to model size and device computing power limitations, requiring multiple network interactions with servers and compromising user experience.

Web frontend, being the closest point to users, can leverage AI capabilities to create better experiences. With rapid hardware advancements, intelligent effects implemented directly in browsers are becoming feasible. Compared to cloud intelligence, edge AI offers advantages of low latency, data privacy protection, and reduced cloud computing resources.

Key browser technologies enabling this include: WebRTC for native video calling and real-time video stream capture; WebXR for GPU-powered complex 3D graphics processing; Web Assembly for bringing other language ecosystems to the web; and Service Workers for offline support and performance improvement. W3C has also established a machine learning community group working on native Web Neural Network API for neural network hardware acceleration.

Web Frontend Use Cases

1. Server Compute Savings and Fast Response : Face tracking and expression recognition applications can be implemented in browsers, requiring real-time screen capture and extremely fast computation for face detection and tracking. Browser implementation ensures universal accessibility without requiring specific app installations.

2. New Interaction Modes : AI capabilities enable complex interactions creating "magical" web experiences. For example, human image segmentation combined with body recognition可以实现非传统的网页交互效果.

3. Privacy Protection : Web AI can perform inference on user data without leaving the phone, protecting privacy. For instance, inappropriate content detection and filtering can be implemented on the client side.

Paddle.js Introduction

Paddle.js is Baidu's JavaScript-based lightweight online inference engine for browser-based web frontends. It leverages Baidu's rich Paddle model library, combining toolchains and online inference libraries to transform models into web-compatible formats at low cost, running in browsers and other web carriers.

The framework has been applied to multiple online projects including face detection, gesture recognition, human segmentation, inappropriate content detection, and Web AR. With only 201KB static code size, it offers excellent execution performance and browser compatibility.

Main Components:

• Model Toolchain : Converts models to browser-friendly formats through format conversion and offline optimization, reducing model size and optimizing performance via operator replacement, fusion, network partitioning, and compression.

• Input/Output Processing : Also called inference pre/post-processing. Paddle.js provides powerful processing capabilities to reduce development costs, allowing frontend engineers to focus on business development.

• Online Inference : Includes neural network loading, online optimization, operator allocation, data initialization, warming, and execution. Uses hardware acceleration technology combined with model quantization compression to ensure real-time execution in browsers.

Simple Usage (4 Steps)

Step 1 - Import:

import Paddle from paddlejs
const paddle = new Paddle();

Step 2 - Load Model:

paddle.load()

Step 3 - Set Input:

const input = io.process({});

Step 4 - Execute and Output:

paddle.execute(input);

Future Outlook

1. Continue optimizing browser execution speed, increasing operator coverage, and extending capabilities to more complex multi-model execution scenarios.

2. Provide more complete user toolchains to further reduce usage costs.

3. Grasp future trends and conduct technical layout for upcoming technologies.

Compatibility

Supports running on browsers with webGL2.0 and webGL1.0 (Chrome, Firefox, Safari, Baidu App, QQ Browser, etc.). Supports NCHW and NHWC format model data computation.