Chloemp/qwen3-4b-text2sql-lora

Qwen3-4B + LoRA, text-to-SQL

LoRA adapter that turns a database schema and a question in English into a SQL query. Trained on a mix of Gretel synthetic SQL and Spider, and selected by a quality gate that requires a proven gain on databases never seen in training.

80.9% execution accuracy on Spider dev, against 75.0% for the un-tuned Qwen3-4B on the same 1,034 examples — a +5.8 point gain, 95% CI [+3.2, +8.4]. In-domain it also gains, +4.5 points [+2.2, +6.8].

Scores are execution accuracy, not string match: both the predicted and the reference query are run against the database and their result sets compared. Two differently-written queries that return the same rows both count as correct.

Results

By Spider difficulty, the gain holds everywhere, and grows with difficulty:

The verdict survives a change of hardware: re-evaluated on an L4 GPU, the same comparison gives +4.6 points in-domain and +5.4 out-of-domain. Greedy decoding flips a handful of examples between machines, so a comparison is only made between evaluations run on the same hardware.

Why the training mix is what it is

An earlier adapter trained on Gretel data alone scored 78.4% in-domain (+4.9) but 65.8% on Spider (−9.3 points). It learned the style of synthetic queries at the cost of general SQL competence — and an in-domain-only evaluation would have shipped it. The quality gate blocked it.

This adapter adds 5,631 Spider training examples, on databases disjoint from Spider dev. It keeps the in-domain gain and beats the base model out-of-domain.

A second negative result is baked into the choice of checkpoint: during training, eval_loss reaches its minimum at step 300 and rises afterwards. That step-300 checkpoint is nevertheless worse on the task — 77.8% against 80.7% on Spider (L4, −2.9 points [−5.0, −0.8]). Loss measures the probability of the reference query; execution accuracy accepts any equivalent query. Pick checkpoints on the task metric, not on the loss.

Usage

The prompt must match training exactly — schema first, then question, with thinking disabled.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

ADAPTER = "Chloemp/qwen3-4b-text2sql-lora"

tok = AutoTokenizer.from_pretrained(ADAPTER)
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, ADAPTER).merge_and_unload().eval()

SYSTEM = ("You translate natural language questions into SQL queries. "
          "Answer with the SQL query only, no explanation.")

schema = """CREATE TABLE singer (singer_id INT, name TEXT, country TEXT, age INT);
CREATE TABLE concert (concert_id INT, singer_id INT, year INT);"""
question = "How many singers are there?"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Database schema:\n{schema}\n\nQuestion: {question}"},
]

text = tok.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tok(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False,
                         pad_token_id=tok.pad_token_id or tok.eos_token_id)

print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# ```sql
# SELECT count(*) FROM singer
# ```

The answer comes back inside a ``` `sql `` block. Strip anything before </think>` first if thinking was left on, then take the fenced block.

enable_thinking=False is not optional: it is what training and evaluation both used, and turning it on changes the output distribution.

Intended use and limits

Written for read-only analytical queries over a schema you supply in the prompt. It was measured on single-table and multi-table SELECTs over small schemas.

• —Never execute its output without guardrails. Asked to delete rows, this model will happily write DELETE FROM singer. The serving stack behind it uses three independent defences: a syntactic whitelist (one read statement, no write nodes, no dangerous functions), a read-only database connection, and a row/time limit. Treat generated SQL as untrusted input.
• —English only, and schemas in the CREATE TABLE form used in training (optionally with a few sample rows).
• —One query per question. No multi-turn correction loop, no use of execution errors to retry.
• —Measured on two suites only. Spider dev covers 20 databases; nothing here says how it behaves on a warehouse-sized schema that does not fit in the prompt.
• —Not compared against a frontier model. The baseline is the un-tuned Qwen3-4B, not GPT-class or Claude-class SQL generation.

Training

LoRA r=16, α=32, dropout 0.05 on all seven projections (q,k,v,o,gate,up,down), loss on the completion only, lr 2e-4 cosine, effective batch 16, 1 epoch, bf16. One A100 80 GB on HF Jobs, about 14 minutes.

Data: 6,096 cleaned Gretel examples (kept only rows whose schema and gold query actually execute under DuckDB) plus 5,631 Spider training examples on databases disjoint from Spider dev. Split fingerprints are recorded so an evaluation can prove which data it ran against.

Training, evaluation and serving import one prompt module, with an assertion that the training prompt is an exact prefix of the evaluation prompt — if the two drift, the measurement is of a different prompt than the one learned.

Evaluation protocol

• —Execution accuracy: both queries run, result sets compared.
• —Row order compared only when the reference query contains ORDER BY; column order ignored (a permutation that makes the results identical counts as a match).
• —Greedy decoding, so runs reproduce.
• —Paired comparison: both models see the same examples, with an exact McNemar test on the discordant ones and a 95% CI on the difference.
• —Promotion requires passing two rules against the current champion: a proven out-of-domain gain (difference > 0, McNemar p < 0.05) and proven in-domain non-inferiority (lower bound of the 95% CI ≥ −2 points). Proving absence of regression is required — a non-significant result is not enough.

License and attribution

Adapter weights follow the base model's Apache-2.0 licence. Training data: `gretelai/synthetic_text_to_sql` (Apache-2.0) and Spider (CC-BY-SA-4.0) — attribute Yale LILY Lab for the latter.