Universal Framework for Joint Image Restoration and 3D Body Reconstruction

Abstract

Recent works have demonstrated excellent state-of-the-art achievements in image restoration and 3D body reconstruction from an input image. The 3D body reconstruction task, however, relies heavily on the input image's quality. A straightforward way to solve this issue is by generating vast degraded datasets and using them in a re-finetuned or newly-crafted body reconstruction network. However, in future usage, these datasets may become obsolete, leaving the newly-crafted network outdated. Unlike this approach, we design a universal framework that is able to utilize prior state-of-the-art restoration works and then self-boosts their performances during test-time while jointly carrying out the 3D body reconstruction. The self-boosting mechanism is adopted via test-time parameter adaptation capable of handling various types of degradation. To accommodate, we also propose a strategy that generates pseudo-data on the fly during test-time, allowing both restoration and reconstruction modules to be learned in a self-supervised manner. With this advantage, the universal framework intelligently enhances the performance without any new dataset or new neural network model involvement. Our experimental results show that using the proposed framework and pseudo-data strategies significantly improves the performances of both scenarios.