SAGE: Segmentation-Aware 3D object extraction from single images

Abstract

Recent progress in vision-based 3D reconstruction has enabled dense point cloud generation directly from a single RGB image, but most existing methods provide only geometric information without semantic context. This limitation hinders object-level understanding and constrains downstream applications such as scene analysis and augmented reality. To address this limitation, we propose a segmentation-aware 3D object extraction framework that combines VGGT, a state-of-to-art geometry transformer, with SegFormer, an efficient semantic segmentation to assign pixel-level category labels while VGGT reconstructs a dense 3D point cloud from the same image. The segmentation results are projected onto the reconstructed points, producing a labeled 3D point cloud where each point is enriched with both geometric and semantic information. Using this representation, we perform clustering within each label by considering point count and density, enabling the segmentation. This approach enables object-level separation directly from single images, allowing labeled 3D reconstructions to be exported as GLB files for visualization and further analysis. Experiments conducted on multiple indoor scenes demonstrate that our system successfully reconstructs point clouds with semantic labels and separates objects into clusters. By unifying semantic segmentation with geometric reconstruction, we propose a robust framework for semantic 3D modeling and object-aware processing.