Pose-Free Face Swapping Based on a Deformable 3D Shape Morphable Model

Yuan LIN  Shengjin WANG  

IEICE TRANSACTIONS on Information and Systems   Vol.E97-D   No.2   pp.305-314
Publication Date: 2014/02/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.E97.D.305
Print ISSN: 0916-8532
Type of Manuscript: PAPER
Category: Computer Graphics
3D face reconstruction,  face swapping,  color transfer,  multi-resolution spline,  human-computer interaction,  

Full Text: PDF(3.3MB)>>
Buy this Article

Traditional face swapping technologies require that the faces of source images and target images have similar pose and appearance (usually frontal). For overcoming this limit in applications this paper presents a pose-free face swapping method based on personalized 3D face modeling. By using a deformable 3D shape morphable model, a photo-realistic 3D face is reconstructed from a single frontal view image. With the aid of the generated 3D face, a virtual source image of the person with the same pose as the target face can be rendered, which is used as a source image for face swapping. To solve the problem of illumination difference between the target face and the source face, a color transfer merging method is proposed. It outperforms the original color transfer method in dealing with the illumination gap problem. An experiment shows that the proposed face reconstruction method is fast and efficient. In addition, we have conducted experiments of face swapping in a variety of scenarios such as children's story book, role play, and face de-identification stripping facial information used for identification, and promising results have been obtained.