anantshri1/nl2sql-agentic-pipeline

NL2SQL Agentic Pipeline

A from-scratch agentic NL→SQL pipeline built with LangChain, Claude, and Voyage AI — evaluated on the BIRD Mini-Dev benchmark.

What this demonstrates

• —Core agent (no prebuilt create_sql_agent): manual NL→SQL chain with a self-correcting retry loop
• —Eval harness: execution-accuracy grading (strict ordered-tuple comparison) on BIRD Mini-Dev, with a 7-category failure taxonomy
• —Schema retrieval: BM25 + Voyage embedding hybrid retrieval via Reciprocal Rank Fusion, benchmarked against full-schema, BM25-only, and embedding-only baselines

Try it

• —`debit_card_specializing` — the core agent + eval harness (Stages A–C). Always uses full schema (5 tables, small enough that retrieval isn't needed).
• —`formula_1` — the retrieval showcase (Stage D). Toggle between full schema, BM25-only, embedding-only, and hybrid (RRF) retrieval strategies, with an adjustable k.

Curated example questions return precomputed results instantly (including a gold-SQL comparison). Free-text questions run the full pipeline live — note these won't include BIRD's "evidence" hints, so live accuracy will typically run lower than the benchmarked numbers below, which were evaluated with evidence.

Key results (formula_1, 66 questions, k=5)

Token cost at k=5: ~55.6% reduction vs. full schema.

See the in-app stats panel for the full k-sweep (k=3/5/7) and the debit_card_specializing baseline (43.3%).

Known limitations (documented, not hidden)

• —BIRD Mini-Dev contains some confirmed gold-SQL labeling defects (flagged during Stage C analysis)
• —Schema retrieval has known failure modes: BM25 vocabulary-mismatch suppression, named-entity grounding gaps (proper nouns/dates not present in 3-row schema samples), and "join-bridge invisibility" (a structurally necessary table with no textual fingerprint matching the question)