Singing Lip Sync Animation System Using Audio Spectrum

Abstract

When humans sing, the changes in pitch and volume are usually significantly greater than when they speak. The shape and size of the mouth changes dramatically when singing compared with when talking, depending on the strength of the voice. In this study, we propose a new model that effectively generates the change in mouth shape in the vocal environment based on the voice strength estimated by analyzing the audio spectrum. We estimate the voice strength by analyzing the spectrum of the voice using a Fast Fourier Transform-based numerical technique for each frame. We apply the intensity of the estimated voice to the morph targets associated with the mouth shape as the weight of the blendshape. Experimental results show a visually convincing lip-synching animation that changes the mouth shape significantly depending on the pitch and volume of the voice.