Introduction to V4L2 and Its Role in Video Codec Development

V4L2 (Video4Linux2) is the Linux kernel framework for video device drivers, originating from the V4L subsystem introduced in Linux 2.1 and upgraded to V4L2 in Linux 2.5. It provides a unified interface for opening, configuring, and streaming video devices.

The V4L2 driver architecture is complex because video hardware often consists of multiple ICs, each exposing separate device nodes under /dev (e.g., /dev/videoX , /dev/vbiX , /dev/radioX ). These ICs are connected via I²C or other buses and are referred to as “sub‑devices”. V4L2 abstracts these components, offering common building blocks that simplify code reuse across drivers.

The framework for video devices includes four main parts:

Character device driver core – V4L2 appears as a character device exposing ioctl interfaces to user space.

V4L2 driver core – provides a standard set of functions for video operations.

Platform V4L2 device driver – registers video_device and v4l2_dev according to platform specifics.

Specific sensor driver – handles power, clock, cropping, streaming, and registers a v4l2_subdev .

V4L2 supports a wide range of device types, including video capture, video output, video overlay, VBI, radio, and codec interfaces.

2. Codec Devices and M2M (Memory‑to‑Memory) Model

Traditional capture devices read data from a camera into DDR memory, while output devices write data from DDR to a display. Codec interfaces differ: they operate in an M2M mode, reading compressed data from DDR, processing it, and writing the result back to DDR. Typical M2M devices are encoders, decoders, scalers, and format converters.

Stateless video codecs do not retain state between frames; the client must maintain and supply the decoding state for each request. This contrasts with stateful codecs where the driver maintains the state.

Interaction between user space and a stateless decoder follows these steps:

Open the device node (e.g., open /dev/videoX ).

Query device capabilities (driver name, supported formats, etc.).

Enumerate and set the desired pixel format.

Request buffers (Android typically uses DMA‑Buf; the kernel creates a videobuf2 queue).

Start streaming with VIDIOC_STREAMON .

Loop: feed each frame to the codec, retrieve processed frames, and return them to user space.

2.1 Buffer Management

Buffer management bridges user space and kernel space for video data (e.g., H.264 streams and decoded YUV frames). Two key structures are:

struct vb2_queue {
    enum v4l2_buf_type type;          // buffer type
    unsigned int io_modes;             // I/O modes (mmap, userptr, etc.)
    const struct vb2_ops *ops;         // queue operation callbacks
    const struct vb2_mem_ops *mem_ops;// memory operation callbacks
    struct vb2_buffer *bufs[VIDEO_MAX_FRAME]; // array of buffers
    unsigned int num_buffers;         // number of allocated buffers
    ...
};

struct vb2_buffer {
    // core buffer fields, including embedded struct v4l2_buffer
};

Buffers can be in three states:

Queued in the driver (filled by the driver, submitted via VIDIOC_QBUF ).

Dequeued to user space (available after VIDIOC_DQBUF ).

Returned to the driver after processing.

V4L2 defines several memory models for buffers:

VB2_MEMORY_MMAP : kernel‑allocated buffers mapped into user space with mmap() . They remain resident in kernel memory.

VB2_MEMORY_USERPTR : user‑allocated buffers whose addresses are passed to the driver; the driver writes directly into user memory.

VB2_MEMORY_DMABUF : buffers shared via DMA‑Buf file descriptors, enabling zero‑copy transfers between user space and kernel (used extensively in Android video pipelines).

3. Summary

The article provides an overview of the V4L2 framework, its role in video codec development, the interaction between user space and stateless decoders, and detailed buffer management mechanisms, highlighting the importance of DMABUF for zero‑copy video processing on Android.

4. References

1. https://www.cnblogs.com/LoyenWang/p/15456230.html 2. https://lwn.net/Articles/203924/ 3. https://www.kernel.org/doc/html/v4.9/media/kapi/v4l2-core.html 4. Android Driver Development Guide (Yang Liu) 5. https://elinux.org/images/e/ec/V4L2-M2M-as-the-driver-framework-for-Video-Processing-IP.pdf