Deformable strokes towards temporally coherent video painting

Abstract

We present an automatic and robust technique for creating non-photorealistic rendering (NPR) and animation, starting from a video that depicts the shape details and follows the motion of underlying objects. We generate NPR from the initial frame of the source video using a greedy algorithm for stroke placements and models, in combination with a saliency map and a flow-guided difference-of-Gaussian filter. Our stroke model uses a set of triangles whose vertices are particles and whose edges are springs. Using a physics-based framework, the generated and rendered strokes are translated, rotated and deformed by forces exerted from the sequential frames. External forces acting on strokes are calculated according to temporally and spatially smoothed per-pixel optical flow vectors. After simulating each frame, we delete unnecessary strokes and add new strokes for disappearing and appearing objects, but only if necessary to avoid popping and scintillation. Our framework automatically generates the coherent animation of rendered strokes, preserving the appearance details and animating strokes along with the underlying objects. This had been difficult to achieve with previous user-guided methods and automatic but limited transformations methods.