Design and Implementation of Bilibili's Self‑Developed Live Streaming P2P System Using WebRTC

01 Introduction

With continuous hardware upgrades, viewers’ computers have become more powerful: faster CPUs, larger and higher‑resolution displays, and widespread fiber‑to‑the‑home networks. These conditions enable higher‑quality video streams.

Audience demand for higher picture quality drives platforms to stream at higher resolutions, which in turn raises bandwidth costs for the platform.

Bandwidth consumption accounts for the largest portion of technical costs in live streaming. To keep costs acceptable, many platforms adopt P2P technology to offload server bandwidth. Cloud providers also offer turnkey P2P solutions for platforms lacking in‑house development capability.

Bilibili’s internal HLS streaming pipeline has matured, providing the prerequisite for a self‑built P2P solution. HLS is a segmented streaming format; each segment is a static file that can be fetched via HTTP Range requests. By having different viewers download different parts of the same segment and then exchange those parts peer‑to‑peer, the server only needs to send each segment once, dramatically reducing bandwidth usage.

02 Using WebRTC for Communication

WebRTC originated from Global IP Solutions, was open‑sourced by Google, and standardized by IETF and W3C. Chrome integrated WebRTC starting with version 28, and other browsers followed. It follows a single standard, allowing cross‑browser connectivity.

WebRTC is the only browser‑native data‑transfer method that does not rely on a traditional client‑server model, making it ideal for P2P bandwidth‑saving scenarios.

2.1 WebRTC Connection Flow

The connection handshake uses the Session Description Protocol (SDP). An SDP Offer is created by the initiator, and the responder returns an SDP Answer. The relevant WebRTC APIs are CreateOffer, SetLocalDescription, CreateAnswer, and SetRemoteDescription. In our case we only use the data channel, so no audio/video tracks are added.

We built a Tracker server that communicates with browsers via WebSocket. The Tracker mediates the exchange of Offer/Answer between viewers watching the same live stream, enabling them to establish direct WebRTC DataChannel connections.

2.2 Application‑Layer Protocol

After the WebRTC DataChannel is established, we transmit binary messages. JSON is unsuitable for binary payloads due to Base64 overhead, so we adopted MessagePack, which is self‑describing, low‑overhead, and natively supports binary data.

The DataChannel uses SCTP. Although SCTP can fragment large messages, browser implementations vary, so we limit each message to 64 KB (practically 60 KB after accounting for MessagePack overhead). This avoids the need for custom fragmentation logic.

To improve throughput, we allow pipelined requests: a client may send a new request before receiving a response to the previous one. Each request carries a unique request ID to match responses.

The protocol works like a stateless HTTP service: a peer A requests a data block from peer B; B replies with the block. If A does not request, B does not push data.

Because HLS segments often exceed 60 KB, each segment (M4S file) is split into multiple 60 KB blocks for P2P transmission. An API is provided for peers to query the download status of each block (downloaded, downloading, not downloaded).

03 Peer Task Allocation

3.1 Task Allocation Challenge

After establishing connections and protocols, the system must orchestrate data exchange to reduce CDN bandwidth. The workflow is:

The live page creates a video player and starts fetching the HLS stream.

Simultaneously, a hidden P2P component connects to the Tracker, announces the stream ID, and receives a list of other viewers.

For each peer, a PeerConnection is created, an SDP Offer is generated, and the Offer is sent via the Tracker. Peers respond with Answers, completing the WebRTC handshake.

Not all connections succeed due to network topology. If the number of successful peers is insufficient, the component requests additional peer lists.

Once connections are established, peers begin downloading needed data blocks from each other according to the protocol described earlier.

3.2 Free‑Market Model

If peers randomly connect, the resulting mesh has heterogeneous edge weights because of “network affinity” (some pairs transfer faster than others). The viewer population is highly dynamic, so a static allocation algorithm would quickly become obsolete.

We therefore let the P2P SDK manage itself under a “free‑market” principle:

Initially, a subset of viewers download distinct portions of the stream directly from the server (seeding).

Peers exchange these blocks; each peer may upload no more than it has downloaded, respecting upload‑download balance.

After the exchange phase, any missing blocks are fetched from the server (completion phase).

The key is to decide which viewers download which blocks. Over‑seeding reduces CDN savings; under‑seeding forces many peers to fetch missing data from the server later. By treating the system as a market, demand (missing blocks) and supply (available blocks) self‑adjust.

When supply exceeds demand for a block, the block is not seeded initially; when demand exceeds supply, the block is prioritized for seeding. To avoid oscillations, peers probabilistically decide whether to switch roles (seed vs. consumer) based on a dice‑roll mechanism. If the market remains unbalanced, another round of probabilistic role changes occurs.

04 Conclusion

The described architecture yields a simple yet efficient live‑streaming P2P system. By constraining each peer’s upload to be no greater than its download and by employing a market‑driven task allocation, the system achieves substantial CDN bandwidth reduction while supporting massive concurrent viewers.

Note: Feedback and discussion are welcome; please like the content if you found it useful.