"Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to Minimize User Input"

AI agent learns when to ask humans for help while navigating, just like a smart intern

This paper introduces a new approach for object navigation where AI agents can ask natural questions to humans during navigation, minimizing the need for detailed upfront instructions. The system uses LLMs and Vision Language Models (VLMs) to enable natural dialogue between agent and user, with uncertainty-aware mechanisms to reduce unnecessary interactions.

-----

https://arxiv.org/abs/2412.01250

🤖 Original Problem:

Current navigation systems require humans to provide complete, detailed object descriptions upfront, which is impractical and burdensome in real-world scenarios.

-----

🔍 Solution in this Paper:

→ The paper proposes AIUTA (Agent-user Interaction with UncerTainty Awareness), a system that enables natural dialogue between agent and human.

→ AIUTA uses a Self-Questioner module that combines VLM and LLM to generate accurate object descriptions while minimizing hallucinations.

→ A novel Normalized-Entropy technique quantifies VLM perception uncertainty, helping filter out unreliable information.

→ The Interaction Trigger module decides when to query humans versus continuing navigation independently.

-----

💡 Key Insights:

→ Template-free, open-ended dialogue is more effective than rigid question-answer formats

→ VLM uncertainty estimation helps reduce hallucinations and improves navigation accuracy

→ Minimizing user interactions while maintaining high success rates is crucial for practical deployment

-----

📊 Results:

→ AIUTA achieves 2x higher success rate than baseline on training split

→ Requires less than 2 questions per successful navigation episode

→ Outperforms state-of-the-art methods on 3 out of 4 benchmark splits