Task-Aware Semantic Map++: Cost-Efficient Task Assignment with Advanced Benchmark

Abstract

Enabling robots to perform diverse tasks autonomously requires a sophisticated semantic understanding of 3D scenes. However, conventional scene representations, which primarily rely on static attributes like visual information or object labels, have significant limitations in allowing robots to infer context-aware actions. We introduce Task-Aware Semantic Map++ (TASMap++), the framework that overcomes these limitations by constructing a map that assigns appropriate tasks to objects based on their holistic context. While prior work like TASMap pioneered this task-centric approach, it suffered from high computational costs and inaccuracies due to its reliance on single-frame analysis, which often fails to capture an object's complete state. In contrast, TASMap++ resolves these issues with a multi-view synthesis pipeline that integrates multiple perspectives of an object for task assignment, resulting in significantly improved computational efficiency over its predecessor. Furthermore, to overcome biases in the existing TASMap evaluation, we established a reliable benchmark derived from the consensus of 32 participants across 231 cluttered scenes. On this benchmark, TASMap++ demonstrates superior accuracy over baselines. Finally, we introduce context-aware grounding, a paradigm distinct from conventional object grounding that relies on visual and spatial attributes. We present a downstream application of TASMap++ as a method to address this challenge and show experimentally that conventional grounding methods struggle in this setting, whereas TASMap++ is markedly more effective. To confirm these findings, the framework's robustness and practicality were validated through extensive experiments on 3D indoor datasets, including real-world scan datasets.