How to Build a Web H.265 Decoder with WebAssembly and FFmpeg

Review

What is H.265?

The article does not re‑introduce H.265; readers can refer to the earlier 2019 post "Web H.265 Player Development Secrets" for details.

WebAssembly Development

Two years after the previous article, WebAssembly 1.1 has been released with many new features and performance improvements, but browsers still do not support H.265 natively. Therefore, playing H.265 in the browser still requires WebAssembly + FFmpeg.

Current Situation

Purpose of This Article

The original H.265 player (Videox.js) has been used in Taobao Live for nearly two years, mainly for B‑side live streaming where low frame rates are acceptable. The new short‑video business targets C‑side users and requires smooth playback of 1080p/720p H.265 streams, as well as support for multiple formats (mp4/fmp4). This article shares the refactored architecture and lessons learned.

Video Demonstration

The following demonstrates the new player playing a 1‑minute 1080p/25fps H.265 MP4 video.

Pre‑load 1,000,000 frames (the whole video) and measure memory usage, CPU usage, and decoding interval.

Because the whole decoding process does not involve playback, the decoding interval equals the time to decode a single frame.

Full decoding of a dozens‑of‑megabyte file consumes up to 4.6 GB of memory and over 300 % CPU on a 4‑core MacBook Pro (2015). The average time to decode one 1080p frame is about 13 ms.

The previous live player needed 26 ms to decode 720p on the same hardware; the new player achieves 13 ms for 1080p, with further optimization possible.

Pre‑load 10 frames and decode them, then feed additional frames gradually to balance decoding and playback.

In this more realistic scenario, average decoding still takes 13 ms per frame, allowing a 25 fps video (40 ms frame interval) to be fed intermittently, reducing CPU usage to around 120 % and memory to about 300 MB while maintaining smooth playback.

Architecture Design

Overall Architecture

The diagram shows the basic skeleton of the new player, consisting of independent modules that exchange data via a common protocol. For example, the Loader passes an ArrayBuffer to the Demuxer, which outputs Packet‑format buffers (Annex‑B) to the Renderer. The Renderer schedules the decoder, handles audio‑video sync, and performs playback. The UI View renders player controls.

Demo Architecture

Because there is no Demuxer, the Loader reads Annex‑B streams directly.

Loader reads Annex‑B stream as a Uint8Array.

postMessage sends the data to a Worker thread where the WASM package decodes it.

WASM calls back with YUV data, which the Worker forwards to the main thread Canvas.

Practical Steps

How to Compile FFmpeg into a WASM Package

First, compile a stripped‑down FFmpeg library. Remove unnecessary modules using the configure script.

1. Preparation

Download the latest Emscripten SDK from the official website.

Download the FFmpeg source (the article uses version 4.1).

emsdk is the tool used to compile FFmpeg into a WASM package.

2. Compile FFmpeg Static Library

Create

make_decoder.sh

with the following content:

<code>echo "Beginning Build:"<br/>rm -r ./ffmpeg-lite<br/>mkdir -p ./ffmpeg-lite # dist directory<br/>cd ../ffmpeg # source directory<br/>make clean<br/>emconfigure ./configure \
  --cc="emcc" \
  --cxx="em++" \
  --ar="emar" \
  --ranlib="emranlib" \
  --prefix=$(pwd)/../ffmpeg-wasm/ffmpeg-lite \
  --enable-cross-compile \
  --target-os=none \
  --arch=x86_32 \
  --cpu=generic \
  --enable-gpl \
  --enable-version3 \
  --disable-swresample \
  --disable-postproc \
  --disable-logging \
  --disable-everything \
  --disable-programs \
  --disable-asm \
  --disable-doc \
  --disable-network \
  --disable-debug \
  --disable-iconv \
  --disable-sdl2 \
  --disable-avdevice \
  --disable-avformat \
  --disable-avfilter \
  --disable-decoders \
  --disable-encoders \
  --disable-hwaccels \
  --disable-demuxers \
  --disable-muxers \
  --disable-parsers \
  --disable-protocols \
  --disable-bsfs \
  --disable-indevs \
  --disable-outdevs \
  --disable-filters \
  --enable-decoder=hevc \
  --enable-parser=hevc<br/>make<br/>make install</code>

Only the HEVC (H.265) decoder is enabled; all other features are disabled to keep the binary small.

Running the script generates a simplified FFmpeg static library and corresponding header files in

ffmpeg-lite

.

3. Write the Entry File

Create

decoder.c

that exposes four functions:

init_decoder

,

decode_buffer

,

decode_packet

, and

output_yuv_buffer

. The file includes the necessary FFmpeg headers and defines a callback type to send YUV data back to JavaScript.

<code>#include <stdio.h><br/>#include <stdlib.h><br/>#include <string.h><br/>#include <libavcodec/avcodec.h><br/>#include <libavutil/imgutils.h><br/><br/>typedef void (*OnBuffer)(unsigned char* data_y, int size, int pts);<br/><br/>AVCodec *codec = NULL;<br/>AVCodecContext *dec_ctx = NULL;<br/>AVCodecParserContext *parser_ctx = NULL;<br/>AVPacket *pkt = NULL;<br/>AVFrame *frame = NULL;<br/>OnBuffer decoder_callback = NULL;<br/><br/>void init_decoder(OnBuffer callback) {<br/>    codec = avcodec_find_decoder(AV_CODEC_ID_HEVC);<br/>    parser_ctx = av_parser_init(codec->id);<br/>    dec_ctx = avcodec_alloc_context3(codec);<br/>    avcodec_open2(dec_ctx, codec, NULL);<br/>    frame = av_frame_alloc();<br/>    frame->format = AV_PIX_FMT_YUV420P;<br/>    pkt = av_packet_alloc();<br/>    decoder_callback = callback;<br/>}<br/></code>

The remaining functions handle buffer parsing, packet decoding, and conversion of

AVFrame

to a contiguous YUV buffer that is sent back via the callback.

Note: Because the build excludes demuxers and bit‑stream filters, decode_buffer must receive raw Annex‑B data.

4. Compile the WASM Package

Create

build_decoder.sh

:

<code>export TOTAL_MEMORY=67108864<br/>export EXPORTED_FUNCTIONS="['_init_decoder','_decode_buffer']"<br/>echo "Running Emscripten..."<br/>emcc decoder.c ffmpeg-lite/lib/libavcodec.a ffmpeg-lite/lib/libavutil.a ffmpeg-lite/lib/libswscale.a \
  -O2 \
  -I "ffmpeg-lite/include" \
  -s WASM=1 \
  -s ASSERTIONS=1 \
  -s LLD_REPORT_UNDEFINED \
  -s NO_EXIT_RUNTIME=1 \
  -s DISABLE_EXCEPTION_CATCHING=1 \
  -s TOTAL_MEMORY=${TOTAL_MEMORY} \
  -s EXPORTED_FUNCTIONS="${EXPORTED_FUNCTIONS}" \
  -s EXTRA_EXPORTED_RUNTIME_METHODS="['addFunction','removeFunction']" \
  -s RESERVED_FUNCTION_POINTERS=14 \
  -s FORCE_FILESYSTEM=1 \
  -o ./wasm/libffmpeg.js<br/>echo "Finished Build"</code>

The resulting

libffmpeg.js

is the JavaScript glue for the WASM module.

How JS Loads and Calls the WASM Module

Worker Part

In the worker thread, the decoder is initialized and used:

<code>export class Decoder extends EventEmitter<IEventMap> {<br/>    M: any;<br/>    init(M: any) {<br/>        this.M = M;<br/>        const callback = this.M.addFunction(this._handleYUVData, 'viii');<br/>        this.M._init_decoder(callback);<br/>    }<br/>    decode(packet: IPacket) {<br/>        const { data } = packet;<br/>        const bufferPtr = this.M._malloc(data.length);<br/>        this.M.HEAPU8.set(data, bufferPtr);<br/>        this.M._decode_buffer(bufferPtr, data.length);<br/>        this.M._free(bufferPtr);<br/>    }<br/>    private _handleYUVData = (start: number, size: number, pts: number) => {<br/>        const u8s = this.M.HEAPU8.subarray(start, start + size);<br/>        const output = new Uint8Array(u8s);<br/>        this.emit('decoded-frame', { data: output, pts });<br/>    }<br/>}</code>

A

DecoderManager

loads the WASM script, handles runtime initialization, caches packets until the module is ready, and forwards decoded frames to the main thread.

<code>class DecoderManager {<br/>    loaded = false;<br/>    decoder = new Decoder();<br/>    cachePackets: IPacket[] = [];<br/>    load() {<br/>        const global: any = self;<br/>        global.Module = { locateFile: (wasm) => './wasm/' + wasm };<br/>        global.importScripts('./wasm/libffmpeg.js');<br/>        global.Module.onRuntimeInitialized = () => {<br/>            this.loaded = true;<br/>            this.decoder.init(global.Module);<br/>            this.push([]);<br/>        };<br/>        this.decoder.on('decoded-frame', this.handleYUVBuffer);<br/>    }<br/>    push(packets: IPacket[]) {<br/>        if (!this.loaded) {<br/>            this.cachePackets = this.cachePackets.concat(packets);<br/>        } else {<br/>            if (this.cachePackets.length) {<br/>                this.cachePackets.forEach(f => this.decoder.decode(f));<br/>                this.cachePackets = [];<br/>            }<br/>            packets.forEach(f => this.decoder.decode(f));<br/>        }<br/>    }<br/>    handleYUVBuffer = (frame) => {<br/>        postMessage({ type: 'decoded-frame', data: frame });<br/>    }<br/>}<br/>const manager = new DecoderManager();<br/>manager.load();<br/>self.onmessage = function(event) {<br/>    const { type, data } = event.data;<br/>    if (type === 'decode') { manager.push(data); }<br/>};</code>

Main Thread Part

The main thread creates the worker, fetches the video stream, and streams chunks to the worker:

<code>import Worker from 'worker-loader!../worker/decoder-manager';<br/>const worker = new Worker();<br/>const url = 'http://example.com/video.hevc';<br/>fetch(url).then(res => {<br/>    if (res.body) {<br/>        const reader = res.body.getReader();<br/>        const read = () => {<br/>            reader.read().then(({ done, value }) => {<br/>                if (!done) {<br/>                    worker.postMessage({ type: 'decode', data: [{ data: value }] });<br/>                    read();<br/>                }<br/>            });<br/>        };<br/>        read();<br/>    }<br/>});</code>

Conclusion

Following the steps above yields a functional H.265 decoder running entirely in the browser via WebAssembly. The same pipeline can be adapted for audio decoding (using the browser's

decodeAudioData

) or other video processing tasks with FFmpeg.

Future articles will cover demuxing MP4/fMP4 streams and rendering YUV data to Canvas.

References

[1] "Web H.265 Player Development Secrets": https://fed.taobao.org/blog/taofed/do71ct/web-player-h265/ [3] WebAssembly: https://webassembly.org/ [4] FFmpeg: https://zh.wikipedia.org/wiki/FFmpeg [7] Emscripten official site: https://emscripten.org/docs/getting_started/downloads.html [8] FFmpeg downloads: https://ffmpeg.org/download.html