"Comprehensive Evaluation for a Large Scale Knowledge Graph Question Answering Service"

This paper introduces Chronos. Chronos is a comprehensive evaluation framework for industrial Knowledge Graph Question Answering services. It addresses the complexities of evaluating multi-component KGQA (Knowledge Graph Question Answering) systems. Chronos focuses on component and end-to-end metrics, scalability, and repeatability.

-----

📌 Chronos dissects Knowledge Graph Question Answering performance at both component and system levels. This modular approach ensures industrial-scale evaluation with precision, exposing weak links in entity linking, relation prediction, and answer selection.

📌 Human-annotated gold labels combined with automated metrics provide a hybrid evaluation strategy. This reduces reliance on noisy heuristics and enables continuous monitoring of degradation across Knowledge Graph Question Answering components.

📌 Error categorization into query understanding and knowledge graph failures offers actionable debugging insights. The dashboard ensures real-time tracking, making Chronos a practical tool for maintaining system reliability at scale.

-----

Paper - https://arxiv.org/abs/2501.17270

Original Problem 😕:

→ Evaluating Knowledge Graph Question Answering systems is challenging.

→ Existing methods often lack comprehensive component-level analysis.

→ Scalability to diverse datasets and repeatability for continuous evaluation are also lacking in current evaluation frameworks.

→ Industrial KGQA (Knowledge Graph Question Answering) systems are complex, with multiple interacting components.

-----

Solution in this Paper 💡:

→ This paper proposes Chronos.

→ Chronos is a modular evaluation framework for KGQA systems.

→ Chronos includes data collection, human annotation, prediction scraping, metrics calculation, and error analysis.

→ Data collection involves using user logs and synthetic data to cover diverse queries.

→ Human annotation provides gold labels for component and end-to-end evaluation.

→ Chronos computes both system-level and component-level metrics.

→ Error analysis categorizes failures into query understanding and knowledge graph errors.

→ A dashboard tracks metrics for continuous monitoring and decision-making.

-----

Key Insights from this Paper 🔑:

→ KGQA systems require component-level and end-to-end evaluation for comprehensive assessment.

→ Diverse datasets and continuous evaluation are crucial for industrial KGQA systems.

→ Human annotation is essential for obtaining high-quality gold labels for evaluation.

→ Automated error analysis and dashboards facilitate debugging and monitoring KGQA system performance.

-----

Results 📊:

→ Chronos was used to evaluate two systems (System 1 and System 2) on 20,000 queries.

→ System 1 achieved 70.91% average E2E precision and System 2 achieved 70.95% average E2E precision.

→ Component-level evaluation showed varying performance across relation prediction, entity linking, and answer prediction.

→ Dataset 3, a challenging dataset, showed reduced performance across all components, highlighting areas for improvement.