High‑Frame‑Rate Video Interpolation: FRUC Algorithm for 25→50 FPS Conversion

During the World Cup, Youku launched a 50‑fps ultra‑clear live stream by converting the standard 25‑fps CCTV feed using a frame‑rate‑up‑conversion (FRUC) algorithm, providing viewers with a smoother and more immersive experience.

High frame rates dramatically improve the perception of fast‑moving objects such as a football, whose speed can exceed 100 km/h. In a 25‑fps video the ball moves more than one meter between frames, resulting in a choppy trajectory, whereas at 50 fps the ball’s path appears continuous and natural.

The FRUC problem is defined as: given the previous frame F(n‑1) and the next frame F(n+1), synthesize the intermediate frame F(n). The basic approach inserts a new frame between every pair of original frames, as illustrated in the diagram below.

Common FRUC methods rely on block‑based motion estimation and interpolation. For each pixel P in the target frame, the algorithm extracts 8×8 blocks around P from both F(n‑1) and F(n+1), finds the most similar block in the opposite frame, and obtains forward (MV_forward) and backward (MV_backward) motion vectors. Assuming uniform motion, the pixel value in F(n) is computed as a weighted average of the two corresponding block pixels after halving the motion vectors.

When the ideal assumptions break (e.g., object deformation, very small or fast objects), block‑based estimation becomes insufficient. To increase precision, the algorithm is refined to use optical flow, which provides per‑pixel motion direction and magnitude. Optical‑flow‑based interpolation yields more accurate intermediate frames, especially for small fast‑moving balls.

A challenging case occurs when the ball becomes too small or fast: the motion vectors point to background pixels, causing the ball to disappear in the interpolated frame. A simple human‑inspired solution is to locate the ball in both neighboring frames, compute the midpoint, and copy the ball’s appearance assuming no deformation. This can be expressed algorithmically by mapping the forward MV/2 from F(n‑1) and the backward MV/2 from F(n+1) to the target position.

Both the block‑based (external) and optical‑flow‑based (internal) methods have strengths: the external method handles rotating scenes better, while the internal method preserves small fast objects. Since most videos contain both scenarios, the final FRUC pipeline fuses the two results.

The fusion consists of two stages: a handcrafted rule‑based merge that fills holes, averages divergent regions, and smooths artifacts, followed by a deep‑learning‑based refinement that combines the handcrafted output with the external frame to produce a natural‑looking result without visible defects.

The FRUC system also supports non‑integer up‑conversion, handling conversions such as 24→50 fps, 25→60 fps, and 30→60 fps, and has been deployed across Youku’s video catalog.