DECO is a framework that helps software teams build and manage enterprise chatbots for engineering tasks, improving incident resolution and documentation access through advanced retrieval methods.
-----
https://arxiv.org/abs/2412.06099
🔍 Original Problem:
Software engineers struggle with scattered documentation, telemetry data, and incident reports across multiple systems, making incident resolution and daily tasks time-consuming and inefficient.
-----
🛠️ Solution in this Paper:
→ DECO introduces a self-hosting model where teams can deploy chatbots without extensive LLM expertise
→ The framework uses NL2SearchQuery functionality and hierarchical planners to optimize document retrieval and skill selection
→ It transforms unstructured incident logs into structured guides through automated preprocessing pipelines
→ The system integrates with internal tools and monitoring systems through an extensible development platform
→ DECO employs a continuous evaluation framework to measure chatbot performance using various metrics
-----
💡 Key Insights:
→ Traditional chatbot development requires iterative prompt tuning, while DECO enables autonomous self-onboarding
→ Response quality issues stem from incorrect skill selection rather than hallucinations
→ Hybrid search algorithms outperform traditional vector search for technical documentation
→ Self-hosting model reduces maintenance costs and resource requirements
-----
📊 Results:
→ Deployed across Microsoft since September 2023 with tens of thousands of interactions
→ Reduces incident triaging time by 10-20 minutes per incident
→ Average user rating of 3.6 stars from hundreds of feedback entries
→ Annual cost savings in tens of millions for the organization
Share this post