Panoptic-FusionNet: Camera-LiDAR fusion-based point cloud panoptic segmentation for autonomous driving

Abstract

Accurate and reliable perception is essential for the safe operation of autonomous cars. The task of Light Detection And Ranging (LiDAR) panoptic segmentation plays a crucial role in predicting point-level classes and instance IDs within a 3D environment, facilitating a comprehensive understanding of the vehicle's surroundings. Existing approaches relying on a single LiDAR sensor often face limitations arising from challenges such as point sparsity and the absence of texture information. To overcome these challenges, the fusion of other sensors, such as cameras, can be employed. However, the development of models for panoptic segmentation utilizing sensor fusion methods remains an unexplored area. In this paper, we present the first LiDAR-camera fusion network for panoptic segmentation named ”Panoptic-FusionNet.” Our proposed network enhances feature maps through a feature fusion module that ensures precise geometric alignment of features extracted from both of sensors. We create tables that have accurately aligned point-voxel-pixel correspondence at multiple scales, and fuse feature maps by querying the table. The fused features are subsequently employed in a bottom-up network to predict semantic labels, instance centers, and offsets. Post-processing is then implemented in a class-adaptive manner, enabling class-wise clustering based on the general sizes of each class, thereby refining the final panoptic results. We evaluate the performance of the proposed network on both the SemanticKITTI and nuScenes datasets, which are large-scale datasets that provide synchronized LiDAR-camera data. In comparison to LiDAR-only baselines, Panoptic-FusionNet attains higher panoptic quality due to its geometrically accurate sensor fusion method, class-adaptive processing, and utilization of instance-augmented pre-trained weights.