Separated Background and Foreground Coding for MPEG Immersive Video Coding

Abstract

In this paper, we propose a new coding method based on the separation of background and foreground data for the MPEG Immersive Video (MIV) coding standard. The main contribution of this work is the development of a background and foreground data separation framework for MIV, along with the integration of spatio-temporal merge techniques to enhance background data reuse. Background data are generally more static than foreground data. By leveraging the relatively static nature of background data, we enable its reuse by applying temporal merge across neighboring frames or spatial merge to convert multi view content into a single view. To extend the functionality of the current MIV, we implemented and made the separated background and foreground coding available and provided the necessary configuration to enable the proposed method. By extending the functionality, it is now possible to encode externally separated foreground and background data using the MIV encoder and decoder. Furthermore, the foreground and background bitstreams can be rendered in one scene or view by applying a spatio-temporal merge to the background. This technique can reduce the bitrate by approximately 5% and the time complexity by approximately 56% compared to the anchor. The proposed method was implemented and adopted in the second edition of the MIV coding standard for the Moving Picture Experts Group (MPEG).