Understanding WebCodecs: Design Goals, Core API, Demos, and Application Scenarios

Author : Liu Jun, Senior Development Engineer at Bilibili.

What is WebCodecs? WebCodecs is a web standard implemented in Chrome 94 (September 2021) that provides direct access to codec capabilities, allowing fine‑grained control over audio and video data.

Problems with Existing Web Audio/Video APIs

Many existing Web APIs indirectly use codecs:

Video playback – Media Source Extensions (MSE)

Audio decoding – WebAudio

Video recording – MediaRecorder

Real‑time streaming – WebRTC

These APIs lack flexible configuration or direct codec access, leading developers to rely on JavaScript or WASM (e.g., ffmpeg.wasm), which suffers from:

Low performance (WebCodecs can be up to 20× faster than ffmpeg.wasm)

High power consumption

Additional network overhead for built‑in codecs

Specific limitations of the older APIs include:

WebAudio can only decode complete audio files, not streams, and provides no encoding.

MediaRecorder records only specific container formats (WebM, MP4) and offers no control over encoding speed or buffers.

WebRTC is tightly coupled with MediaStream and is opaque, suitable only for real‑time communication.

Video tag and MSE are great for playback but cannot control decode rate, buffer length, or support many container formats.

Design Goals of WebCodecs

Streaming : Stream input/output from remote or disk resources.

Efficiency : Leverage hardware acceleration and run in Web Workers.

Composability : Work well with Streams, WebTransport, WebAssembly, etc.

Recoverability : Ability to recover from network issues or resource shortages.

Flexibility : Adapt to hard‑real‑time, soft‑real‑time, and offline scenarios.

Symmetry : Similar patterns for encoding and decoding.

What WebCodecs Does NOT Target

Container (muxing/demuxing) APIs.

Implementing codecs purely in JS or WASM.

These points are summarized from the translated WebCodecs explainer.

What WebCodecs Can Do

The API sits in the video production‑consumption pipeline, providing two key capabilities:

Control over the encoding/decoding process.

Access to raw binary data before and after codec operations.

Key types include VideoFrame , EncodedVideoChunk , VideoEncoder , and VideoDecoder . All image‑related Web types can be converted to VideoFrame , and the API design emphasizes symmetry between encoding and decoding.

WebCodecs does not handle:

Capture/render of audio‑video data.

Container muxing/demuxing.

Transport, storage, or other pipeline stages.

Related Web APIs

WebAudio – audio processing.

WebGPU / WebGL + OffscreenCanvas – off‑screen image processing.

OPFS – local file read/write.

WebWorker + WASM – heavy computation.

WebTransport – low‑latency network transport.

Combining these with WebCodecs enables powerful media applications.

Demo Implementations (using WebAV, a SDK built on WebCodecs)

1. Controllable Decoding

import { MP4Clip } from '@webav/av-cliper'

// Initialize with an MP4 stream
const clip = new MP4Clip((await fetch('
')).body)
await clip.ready
let time = 0
// Render all frames as fast as possible
while (true) {
  const { state, video: videoFrame } = await clip.tick(time)
  if (state === 'done') break
  if (videoFrame != null && state === 'success') {
    ctx.clearRect(0, 0, cvs.width, cvs.height)
    // Draw to Canvas
    ctx.drawImage(videoFrame, 0, 0, videoFrame.codedWidth, videoFrame.codedHeight)
    // Release the frame immediately
    videoFrame.close()
  }
  // Time unit is microseconds; ~30 fps
  time += 33000
}
clip.destroy()

2. Adding Watermark

Convert text to an image, animate its position, and composite with the video.

const spr1 = new OffscreenSprite(new MP4Clip((await fetch('
')).body))
const spr2 = new OffscreenSprite(new ImgClip('Watermark'))
spr2.setAnimation(/* animation config */)
const com = new Combinator()
await com.add(spr1, { main: true })
await com.add(spr2, { offset: 0 })
// com.output() => output video stream

3. Green‑Screen Chroma‑Key

// Create chroma‑key helper
const chromakey = createChromakey(/* config */)
const clip = new MP4Clip((await fetch('
')).body)
clip.tickInterceptor = async (_, tickRet) => {
  if (tickRet.video == null) return tickRet
  return { ...tickRet, video: await chromakey(tickRet.video) }
}

4. Bloom‑Shadow (video recording tool)

Provides a browser‑based video capture & editing workflow for courses, live streaming, etc.

Application Scenarios

Video production: encoding, trimming, watermarking, thumbnail generation.

Video consumption: fine‑grained playback control, low‑latency streaming, frame‑by‑frame playback.

Compute off‑loading: move encoding/compression tasks to the client, reducing server cost and latency.

Advantages

Performance : WebCodecs can be ~20× faster than ffmpeg.wasm for simple encoding tasks.

Web Platform Benefits : Cross‑platform, rapid iteration, and increasingly mature low‑level APIs.

Limitations

Immature ecosystem – limited container tools, fewer third‑party libraries.

Browser compatibility – older browsers lack support.

Codec availability is constrained by what browsers ship.

Configurable codec parameters are less extensive than native libraries.

Custom codec implementation is not yet possible.

These constraints may be mitigated over time as the community contributes more tools.

Vision

WebCodecs becomes the foundational audio‑video processing layer for the web.

Like HTML5 did for graphics and layout, WebCodecs will accelerate media‑centric web applications.

Appendix

WebAV SDK (https://github.com/hughfenghen/WebAV)

Translated WebCodecs explainer (https://hughfenghen.github.io/posts/2023/10/02/webcodecs-explainer/)

Web Audio‑Video Overview (https://hughfenghen.github.io/posts/2023/07/16/webav-0-overview/)

WebAV demo (https://hughfenghen.github.io/WebAV/demo/1_1-decode-video)

Bloom‑Shadow project (https://github.com/hughfenghen/bloom-shadow)