A Framework for Virtual 3D Manipulation of Face in Video

Abstract

This paper presents a framework that enables a user to manipulate his/her face shape three-dimensionally in video. Existing face manipulation applications and methods have some limitations: single photo, manipulation on image domain, or limited deformation. In the proposed framework, face is tracked from video by using landmark tracking and fitting 3d morphable face model to image, and the face model is further deformed according to the user input with mesh deformation method and rendered with texture from the frame image onto the camera preview. Therefore, unlike conventional applications and researches for face manipulation, the proposed framework allows the user to perform free-form 3d deformation of face in video and to view the deformed face at various viewpoints.