DECO is a framework that helps software teams build and manage enterprise chatbots for engineering tasks, improving incident resolution and documentation access through advanced retrieval methods.
-----
https://arxiv.org/abs/2412.06099
π Original Problem:
Software engineers struggle with scattered documentation, telemetry data, and incident reports across multiple systems, making incident resolution and daily tasks time-consuming and inefficient.
-----
π οΈ Solution in this Paper:
β DECO introduces a self-hosting model where teams can deploy chatbots without extensive LLM expertise
β The framework uses NL2SearchQuery functionality and hierarchical planners to optimize document retrieval and skill selection
β It transforms unstructured incident logs into structured guides through automated preprocessing pipelines
β The system integrates with internal tools and monitoring systems through an extensible development platform
β DECO employs a continuous evaluation framework to measure chatbot performance using various metrics
-----
π‘ Key Insights:
β Traditional chatbot development requires iterative prompt tuning, while DECO enables autonomous self-onboarding
β Response quality issues stem from incorrect skill selection rather than hallucinations
β Hybrid search algorithms outperform traditional vector search for technical documentation
β Self-hosting model reduces maintenance costs and resource requirements
-----
π Results:
β Deployed across Microsoft since September 2023 with tens of thousands of interactions
β Reduces incident triaging time by 10-20 minutes per incident
β Average user rating of 3.6 stars from hundreds of feedback entries
β Annual cost savings in tens of millions for the organization
