Integrating Video Rendering into the Cocos Engine: Multi‑threaded OpenGL and Audio/Video Pipeline on Android

Let Cocos Engine Directly Render Video

Instead of using native Android views such as SurfaceView or TextureView, the team chose to render video inside the Cocos engine to maintain precise layering and animation control between video and game elements.

Advantages of native mechanisms: direct support for standard player interfaces, stable rendering performance, and simple code.

Disadvantages of native mechanisms: inability to finely adjust video‑game layering and performance overhead when animating the player from the game side.

Therefore, a custom widget based on cocos2d::ui::Widget was created; its draw callback receives video frames converted to a texture and renders them via a cocos2d::Sprite using a cocos2d::Texture2D supplied by the widget.

Implementation Details

The native layer converts video output into a texture, passes it to the widget, which then creates a Texture2D for the sprite. The following diagram illustrates the conversion chain:

Multithreaded OpenGL

Cocos runs on its own thread with a dedicated OpenGL context. Performing video‑related OpenGL operations on this context can corrupt the engine’s state, so all audio/video processing must occur on separate worker threads.

To share textures across threads, a common ShareGroup is required. On Android the team replaced the default EGL context factory in Cocos2dxGLSurfaceView::setEGLContextFactory with a custom implementation that creates contexts sharing the same group, and applied the same approach to the video processing modules.

Audio/Video Processing Pipeline Model

Model Construction

The pipeline abstracts the workflow into three parts: an audio/video source, a thread dispatcher, and a chain of consumers. The source loads or captures video frames, the dispatcher routes data to appropriate worker threads, and each consumer processes the data, with the final consumer often being a custom Cocos widget.

Audio/Video Frame Data Reuse

To avoid frequent memory allocations, a ByteBuffer pool is used. Buffers are reused only when the pool buffer size meets or exceeds the required size. Each buffer carries a tag indicating its resolution, enabling quick lookup for matching frames.

Recycling is handled via a close‑able reference mechanism similar to Fresco’s CloseableReference : the reference count drops to zero only after all threads have released the buffer, at which point it is returned to the pool.

Work Thread Blocking Monitoring

On low‑end devices, task queues can become saturated, causing audio‑video desynchronization. The solution uses a HandlerThread combined with a manageable queue. Tasks are enqueued, and the handler thread dequeues them for processing.

The queue records metrics such as expected execution time, actual processing time, and queue length, allowing the system to drop tasks strategically or downgrade video resolution when necessary.

Conclusion

The described techniques—direct video rendering in Cocos, shared OpenGL contexts across threads, and a modular audio/video pipeline—have been successfully deployed in the Liulishuo child‑learning app, delivering stable performance and extensibility, and paving the way for upcoming live‑class features.