"Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL"

Is more context really better? Yes, for NL2SQL (Natural Language to SQL) accuracy, definitively.

This paper explores how long context LLMs can improve NL2SQL performance by leveraging extended context windows and additional contextual information, achieving strong results without fine-tuning or self-consistency methods.

-----

Paper - https://arxiv.org/abs/2501.12372

Original Problem 😞:

→ NL2SQL (Natural Language to SQL) is challenging due to natural language ambiguity and the need for precise SQL generation.

→ Existing NL2SQL pipelines struggle with schema linking and semantic understanding.

→ Prior approaches often rely on complex prompts, chain-of-thought reasoning, and in-context learning with limited context windows.

-----

Solution in this Paper 💡:

→ This paper investigates the use of extended context windows in LLMs for NL2SQL (Natural Language to SQL).

→ It explores various contextual information types: column examples, question-SQL pairs, user hints, SQL documentation, and schema.

→ The paper proposes a long-context NL2SQL pipeline using Google's gemini-1.5-pro.

→ This pipeline includes stages for generation, fixing/rewriting, and verification using agentic workflow calls.

→ It uses all database tables to ensure high recall in schema linking.

→ It incorporates column descriptions and sample values for improved column selection.

→ It leverages user-provided hints for clarification.

→ It uses synthetic examples for many-shot in-context learning.

-----

Key Insights from this Paper 🤔:

→ 100% recall of relevant tables and columns in context is crucial for high-quality SQL generation.

→ Long context models are robust and do not get lost with irrelevant table information.

→ Synthetically generated examples, based on SQL structure similarity, enhance performance more than examples based on question similarity.

→ Hints significantly improve accuracy, followed by column sample values and self-correction.

→ SQLite documentation inclusion does not significantly boost accuracy.

→ Latency increases linearly with context size, presenting a trade-off between accuracy and speed.

-----

Results 📊:

→ Achieves 67.41% execution accuracy on the BIRD benchmark (dev) without fine-tuning.

→ Outperforms E-SQL (65.58%) and MCS-SQL (63.36%) without fine-tuning or self-consistency.

→ Demonstrates competitive performance compared to fine-tuned and self-consistency based methods.