Framer: Interactive Video Frame Interpolation Using Diffusion Models

Video frame interpolation inserts new frames between existing ones to increase frame rate, improving smoothness and quality.

Framer, proposed by Zhejiang University and Ant Research, is an interactive video frame interpolation method based on a large‑pretrained video diffusion model. It generates smooth transitions between two images guided by user‑defined trajectories or an automatic mode that estimates keypoints.

The method enhances traditional optical‑flow approaches by using a Co‑Tracker algorithm to obtain robust keypoint trajectories, converting keypoint coordinates into Gaussian heatmaps for a control module, and adding a control‑network branch after the encoder to inject trajectory features into the decoder.

Framer supports custom trajectories, automatic keypoint estimation, and can handle complex scenes via cross‑frame correspondence. Experiments show strong performance in image deformation, video generation, cartoon interpolation, and converting sketches to videos.

For more details see the paper https://arxiv.org/abs/2410.18978 and try the online demo on HuggingFace https://huggingface.co/spaces/wwen1997/Framer . The code will be released on GitHub https://github.com/aim-uofa/Framer .

The authors are from Prof. Shen Chunhua’s team at Zhejiang University and Ant Technology Research Institute’s Interactive Intelligence Lab.