Looking beyond input frames: Self-supervised adaptation for video super-resolution

Abstract

Recent test-time adaptive video super-resolution (VSR) methods have elevated the performance by exploiting the self-similar patches within the low-resolution (LR) frames to adapt to given input frames. However, the LR frames contain a limited amount of such patches, limiting the performance of such adaptation methods, especially for a challenging scale factor (e.g., ×4). In this work, we propose to explore beyond the input LR frames. In particular, we observe that a greater amount of self-similar patches across various scales can be found from estimated high-resolution (HR) frames (i.e., initially restored frames) produced by a pre-trained VSR network. Upon the observation, we propose a new self-supervision test-time adaptation approach via self-distillation to exploit such rich amount of self-similar patches from initially restored frames. Specifically, we perform self-distillation by exploiting multi-scale relationship: distilling knowledge from larger patches to smaller ones with similar semantics. Our framework is flexible and effective as the knowledge can be distilled either from the network itself or the larger one. Furthermore, our framework demonstrates the robustness, being able to recover from undesirable artifacts present in initially restored frames. Extensive evaluation with various VSR networks on numerous datasets reveals that our algorithm consistently improves the restoration quality by a large margin without ground-truth HR video frames. Code is available at: https://github.com/jinsuyoo/bissa.