MV-HEVC and Its Application in 3D Video Encoding for Apple Vision Pro

On June 6, 2023, Apple announced the Vision Pro at WWDC, highlighting hardware support for MV‑HEVC (Multi‑View HEVC) to significantly improve the subjective and objective experience of 3D video. This article explains what MV‑HEVC is, how it differs from traditional HEVC, and its relationship with Apple Vision Pro and Tencent's V265 encoder.

3D Imaging Techniques Overview

Common 3D video technologies include holographic projection, glasses‑free (裸眼) 3D displays, stereoscopic cinema, and VR head‑sets. Holographic projection creates true 3‑D images in air but is still experimental. Glasses‑free 3D uses special gratings or lenses to deliver slightly different images to each eye. Stereoscopic cinema relies on polarized or color‑filtered glasses. VR head‑sets, such as Meta Oculus and Apple Vision Pro, present separate left‑ and right‑eye views to achieve immersion.

Current 3D Video Encoding Methods

Most 3D content (VR, stereoscopic movies) is encoded using the side‑by‑side (SBS) format, where left and right eye frames are packed into a single frame and then encoded with a conventional video codec such as HEVC. This approach does not exploit inter‑view redundancy, leading to sub‑optimal compression.

MV‑HEVC Standard

In 2014, the JCT‑3V group extended HEVC to support multi‑view video, creating the MV‑HEVC standard. It introduces a new NALU syntax element LayerId to identify the view (LayerId 0 for the primary view, LayerId 1 for the secondary view). Frames sharing the same picture order count (POC) but belonging to different layers form an Access Unit (AU). MV‑HEVC adds inter‑layer reference frames, allowing the secondary view to reference the primary view’s frame, thus reducing inter‑view redundancy.

Inter‑Layer Mode and TMVP

MV‑HEVC defines an Inter‑Layer prediction mode that extends HEVC’s temporal motion vector prediction (TMVP) across layers. However, when the reference and current frames belong to the same POC, scaling factors can become zero, causing division‑by‑zero errors. To avoid this, MV‑HEVC mandates that all inter‑layer references are marked as long‑term reference frames, which are only used for inter‑layer prediction.

Encoder and Decoder Support

Apple Vision Pro’s chipset already provides hardware decoding for MV‑HEVC streams. Tencent’s V265 encoder has been extended to support MV‑HEVC, integrating the new syntax and reference‑frame search algorithms without sacrificing speed. Experimental results on 3D movie test cases show over 50 % bitrate savings for the secondary view (layer 1) and more than 20 % overall savings when both views are encoded together, translating to roughly a 20 % reduction in bandwidth for the same visual quality.

Conclusion

By adding MV‑HEVC extensions, V265 gains native 3D video encoding capabilities, achieving up to 20 % bandwidth reduction at equal quality. Combined with the hardware‑friendly nature of MV‑HEVC, this advancement will continue to drive the development of immersive 3D video applications.