MAMFI: Learning Motion at All Scales via Memory-as-Motion Attention for Frame Interpolation

Abstract

We propose MAMFI, a hybrid architecture for video frame interpolation that effectively captures both large-scale and fine-grained motion. Prior approaches often suffer from a seesaw effect, where improving performance for large motions degrades accuracy for small motions, and vice versa. To address this, MAMFI introduces a novel architecture that combines two complementary motion memory modules with cross-sliding window attention (X-SWA). The motion memory modules efficiently store large-scale motion and task-level prior knowledge, while the X-SWA mechanism precisely captures dense, local motion between adjacent frames. Furthermore, we incorporate test-time online learning, enabling the model to adaptively update memory in regions with significant motion while discarding redundant information. As a result, MAMFI achieves superior performance over state-of-the-art methods by modeling motion across multiple scales with both accuracy and flexibility, as demonstrated on several benchmark datasets.