+
+ Existing embodied instance goal navigation tasks, driven
+ by natural language, assume human users to provide
+ complete and nuanced instance descriptions prior to the
+ navigation, which can be impractical in the real world as
+ human instructions might be brief and ambiguous.
+
+
+ To bridge this gap, we propose a new
+ task, Collaborative Instance Navigation (CoIN), with
+ dynamic agent-human interaction during navigation to
+ actively resolve uncertainties about the target instance
+ in natural, template-free, open-ended dialogues.
+
+
+ To address CoIN, we propose a novel
+ method, Agent-user Interaction with UncerTainty Awareness
+ (AIUTA), leveraging the perception capability of Vision
+ Language Models (VLMs) and the capability of Large
+ Language Models (LLMs). First, upon object detection, a
+ Self-Questioner model initiates a self-dialogue to obtain
+ a complete and accurate observation description, while a
+ novel uncertainty estimation technique mitigates
+ inaccurate VLM perception. Then, an Interaction Trigger
+ module determines whether to ask a question to the user,
+ continue or halt navigation, minimizing user input.
+
+
+
+ For evaluation, we introduce CoIN-Bench,
+ a benchmark supporting both real and simulated humans.
+ AIUTA achieves competitive performance in instance
+ navigation against state-of-the-art methods, demonstrating
+ great flexibility in handling user inputs.
+
+