Consistent Object Removal from Masked Neural Radiance Fields by Estimating Never-Seen Regions in All-Views

Abstract

Neural radiance field (NeRF) is a technique for synthesizing novel-view images based on an understanding of scene geometry. Recently, there have been studies that remove objects from NeRF, which makes it possible to synthesize novel-view images with objects removed. Most existing methods apply a pretrained inpainting model to each multi-view image to remove objects, and use these images to train the NeRF model. However, these approaches not only require a lot of feedforward of the inpainting model, but also lead to inconsistency problems between the inpainted images. To address these limitations, we propose a method to minimize the areas that need to be filled. To this end, we estimate never-seen regions that are occluded in all images based on density, and apply inpainting only to those regions. After removing target objects, we select the images that allow the final trained NeRF to consistently fill in the removed regions. Therefore, the proposed method consistently removes target objects from NeRF, and the effectiveness of the proposed method is demonstrated through various experiments. Furthermore, we suggest practical techniques to simplify the training processes and provide a new 360∘ real-world dataset for inpainting in NeRF.