How Frontend AI Is Evolving: From WebGL to MNN.js

AI in the Frontend During the Pandemic

AI has been booming in the large‑frontend space for two to three years. With the rise of device compute and on‑device models, scenarios ranging from image classification to facial effects, voice recognition, and even autonomous driving are emerging. During the pandemic, AI entered daily life through virus research and case tracking, and frontend developers created a mask‑overlay WebApp that went viral on Zhihu and social circles.

Why Is Frontend AI Less Visible Than Native?

Since 2017, frameworks providing low‑level acceleration for mobile apps have proliferated. In the frontend, options are limited mainly to TensorFlow JS, with ONNX JS and WebNN still maturing.

Framework Landscape

TensorFlow JS remains the primary choice, but its updates lag behind native libraries. ONNX JS suffers from infrequent releases, making it a less reliable option.

WebNN selects the best platform‑provided accelerator (DirectML, NNAPI) and falls back to WebGL, WebGPU, or WebAssembly when needed. If adopted as a browser standard, it could enable low‑cost compute acceleration, though platform fragmentation limits feature coverage.

Introducing MNN.js

Leveraging experience from mini‑program optimization and MNN GPU tuning, MNN.js offers a new JavaScript inference engine with significant performance gains. Tests on desktop (MacBook Pro), iOS (iPhone X), and Android (Xiaomi 9) show up to 70% faster first‑inference time and over 35% reduction in subsequent runs, with 4× higher frame rates on mid‑range devices.

Key features:

Supports 38 operators and many complex models.

Reuses MNN’s model format, enabling conversion from Caffe, TensorFlow, and PyTorch.

Three backends: WebGL, WebAssembly, and JavaScript (the latter rarely used).

Example loading and running a model:

<code>async function loadModel() {
  let compiled = await MNN.WasmBuilder.compile(fetch('/path/to/mnn.wasm'));
  let wasm = await MNN.WasmBuilder.create(compiled);
  let js = MNN.JSBuilder.create();
  let builders = [webgl, wasm, js];
  let result = await fetch(url);
  let model = await result.arrayBuffer();
  return MNN.loader.loadMNN(new Uint8Array(model), builders);
}

async function run(webcam) {
  let width = webcam.videoWidth;
  let height = webcam.videoHeight;
  let preprocess = {
    sourceFormat: 'rgba',
    destFormat: 'bgr',
    filterType: 'nearest',
    wrapType: 'zero',
    cropAndResize: [[0,0,width,height],[0,0,224,224]],
    means: [103.94,116.78,123.68],
    norms: [1/127.5]
  };
  let rgba = capture(webcam);
  await model['data'].setData(rgba, "uint8", [1,height,width,4], "NHWC", preprocess);
  let result = await model['prob'].getData("NHWC");
  let values = new Float32Array(result.data);
  let sorted = values.slice().sort((a,b)=>b-a);
}
</code>

The API retains MNN’s dynamic graph design and provides image preprocessing, ensuring stable performance even in video scenarios.

Where Is Frontend AI Heading?

Deep learning relies on data, algorithms, and compute. While data and models improve, compute constraints still limit practical AI applications. Frontend AI must bridge business needs with algorithmic possibilities, especially as mini‑programs become a universal delivery platform, offering new user‑engagement opportunities.

Taobao’s shopping mini‑program exemplifies this trend, integrating AI/AR features such as face, pose, and gesture recognition to create interactive shopping experiences.

Details and recruitment can be found at open.m.taobao.com.

Call for Collaboration

MNN.js is not yet open‑source; contributors interested in frontend AI are welcome to join the MNN DingTalk group for further discussion.