adelelsayed1991/fhirsql-reasoning-sql

FHIR-to-SQL with Database-Resolved Clinical Terminology

Natural-language hospital questions paired with a structured JSON query plan and compiled DuckDB SQL, over a FHIR-derived clinical schema. Built entirely from synthetic Synthea patients — no real patient data.

Companion to the paper Plan-Then-Compile: Turning a General-Purpose Coder Model into a FHIR Data Analyst.

• —Code & paper: https://github.com/adelelsayed/fhirsql-reasoning-sql
• —Adapters: https://huggingface.co/adelelsayed1991/fhirsql-reasoning-sql-adapters

What makes this dataset different

Gold SQL never embeds a literal clinical code. Every query resolves its concept at runtime through a lookup against a valuesets terminology table:

WITH resolved AS (
    SELECT code, code_system FROM valuesets
    WHERE table_name = 'procedure' AND display ILIKE '%Depression screening (procedure)%'
)
SELECT DISTINCT patient_id
FROM procedure, resolved
WHERE procedure.code = resolved.code AND procedure.system = resolved.code_system
  AND status = 'completed'

This turns terminology resolution from a memorization problem into a queryable one, and makes generalization to clinical concepts absent from training measurable.

Splits

Held-out splits come from a disjoint 6,383-patient population (training: 18,999 patients; zero patient_id overlap).

• —`heldout_familiar` — clinical concepts also used in training, different patients. Isolates population generalization. 96.5% of its answerable rows are byte-identical `(question, gold_sql)` pairs from `train`, so it measures whether learned queries stay correct on new data; it cannot distinguish memorization from generalization.
• —`heldout_unseen` — 84 clinical concepts appearing nowhere in training (verified by set intersection; zero shared question/query pairs). This is the split for measuring transfer.

⚠️ The 1,016 abstention rows are the same rows in all three splits. Unanswerable questions reference no patient data, so an identical set was reused. Abstention metrics computed on the held-out splits therefore measure retention of trained refusal behavior, not held-out refusal generalization.

⚠️ Execution match is weakly discriminative on `heldout_unseen`. Its concepts are rare (1–3 patients each) and gold answers are small integers: 53.9% of within-archetype concept pairs return identical results, so a wrong concept can go undetected. Text-match metrics are not subject to this.

Fields

Provenance

Every answerable row's gold SQL was executed against the real database and kept only if it ran without error and returned at least one row, then checked for repeated-execution stability. Questions come from 160 role-appropriate phrasings authored with LLM assistance under the author's direction, then parameterized by concept.

schema.sql is the exact schema injected into every prompt. valuesets_ddl.sql is an ablation fragment that was not part of the prompt during the published runs — see the paper, §5.6.

Limitations

Synthetic data only; one question-generation distribution (templated phrasing, not free-form clinician language); terminology resolution is substring matching against a corpus-derived dictionary with one display string per code, so synonymy, hierarchy, and ambiguity are out of scope (ambiguous concepts are excluded from gold). Some regulatory questions say "dispensed" but are answered from MedicationRequest (prescription orders). Full discussion in the paper's §7.

Citation

See CITATION.cff in the GitHub repository.