Video Inpainting by Frame Alignment with Deformable Convolution

Yusuke HARA
Xueting WANG
Toshihiko YAMASAKI

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E104-D    No.8    pp.1349-1358
Publication Date: 2021/08/01
Publicized: 2021/04/22
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2020EDP7194
Type of Manuscript: PAPER
Category: Image Processing and Video Processing
Keyword: 
video inpainting,  deformable convolution,  deep learning,  computer vision,  

Full Text: PDF>>
Buy this Article


Summary: 
Video inpainting is a task of filling missing regions in videos. In this task, it is important to efficiently use information from other frames and generate plausible results with sufficient temporal consistency. In this paper, we present a video inpainting method jointly using affine transformation and deformable convolutions for frame alignment. The former is responsible for frame-scale rough alignment and the latter performs pixel-level fine alignment. Our model does not depend on 3D convolutions, which limits the temporal window, or troublesome flow estimation. The proposed method achieves improved object removal results and better PSNR and SSIM values compared with previous learning-based methods.