Video Frame Interpolation via Residue Refinement

Video frame interpolation achieves temporal super-resolution by generating smooth transitions between frames. Although great success has been achieved by deep neural networks, the synthesized images stills suffer from poor visual appearance and unsatisfactory artifacts. In this paper, we propose a novel network structure that leverages residue refinement and adaptive weight to synthesize in-between frames. The residue refinement technique is used for optical flow and image generation for higher accuracy and better visual appearance, while the adaptive weight map combines the forward and backward warped frames to reduce the artifacts. Moreover, all sub-modules in our method are implemented by U-Net with less depths, so the efficiency is guaranteed. Experiments on public datasets demonstrate the effectiveness and superiority of our method over the state-of-the-art approaches.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Video Frame Interpolation MSU Video Frame Interpolation RRIN PSNR 25.76 # 19
SSIM 0.893 # 20
VMAF 59.82 # 20
LPIPS 0.072 # 22
MS-SSIM 0.902 # 20
Video Frame Interpolation UCF101 RRIN PSNR 34.93 # 16
SSIM 0.9496 # 15
Video Frame Interpolation Vimeo90K RRIN PSNR 35.22 # 13
SSIM 0.9643 # 17

Methods