mahboobalam0/distill-text2sql

Distill — Fine-tuned Qwen2.5-3B for Text-to-SQL

Distill is a QLoRA fine-tuned LoRA adapter based on `Qwen/Qwen2.5-3B-Instruct`, specialized for natural-language-to-SQL generation.

The model takes a database schema and a natural-language question and generates a SQL query.

The model was fine-tuned on the sql-create-context dataset using 4-bit QLoRA on a single NVIDIA T4 GPU. The final LoRA adapter is approximately 114 MB with approximately 21M trainable parameters.

Model Details

• —Developed by: MahboobAlam0
• —Model type: LoRA / QLoRA adapter
• —Base model: Qwen/Qwen2.5-3B-Instruct
• —Task: Natural language → SQL
• —Fine-tuning method: QLoRA
• —Dataset: sql-create-context
• —Training examples: 28,000
• —Validation examples: 7,858
• —LoRA rank: 16
• —LoRA alpha: 32
• —Quantization: 4-bit NF4 with double quantization
• —Trainable parameters: ~21M
• —Adapter size: ~114 MB
• —Hardware: 1 × NVIDIA T4
• —Training platform: Kaggle
• —Programming language: Python 3.12

Model Description

Distill adapts Qwen2.5-3B-Instruct for text-to-SQL generation using parameter-efficient fine-tuning.

The base model is loaded using 4-bit quantization and kept frozen while LoRA adapters are trained on selected attention and MLP projection layers.

The LoRA adapters target:

• —q_proj
• —k_proj
• —v_proj
• —o_proj
• —gate_proj
• —up_proj
• —down_proj

The model is designed to generate SQL from a database schema and a natural-language question.

Architecture

Natural Language Question
          +
    CREATE TABLE Schema
          │
          ▼
┌─────────────────────────────────────┐
│ Qwen2.5-3B-Instruct                 │
│ Frozen 4-bit NF4 quantized weights │
│                                     │
│ LoRA adapters (~21M parameters)     │
│                                     │
│ Attention: q, k, v, o              │
│ MLP: gate, up, down                │
└─────────────────────────────────────┘
          │
          ▼
      Generated SQL

Intended Use

This model is intended for:

• —Natural-language database querying
• —Text-to-SQL applications
• —SQL generation
• —Database assistants
• —Analytics interfaces
• —Business intelligence prototypes
• —SQL education and experimentation
• —Research into parameter-efficient fine-tuning

A typical input contains:

1. 1.A SQL database schema
2. 2.A natural-language question

The model then generates the corresponding SQL query.

Example

Schema

CREATE TABLE orders (
    id INTEGER,
    customer VARCHAR,
    amount FLOAT,
    status VARCHAR
);

Question

What is the total revenue from completed orders?

Generated SQL

SELECT SUM(amount)
FROM orders
WHERE status = 'completed';

Out-of-Scope Use

This model should not be treated as an unrestricted autonomous database agent.

It should not be used without appropriate validation for:

• —Destructive database operations
• —Production database migrations
• —Database authorization or access-control decisions
• —Security-sensitive operations
• —Financial decisions
• —Medical decisions
• —Legal decisions
• —Other high-stakes applications

Generated SQL should always be treated as untrusted model output.

Limitations

Dataset limitations

The model was trained on sql-create-context, which primarily contains single-table schema examples.

Performance may therefore decrease on:

• —Complex multi-table schemas
• —Multiple joins
• —Nested queries
• —Large enterprise schemas
• —Highly domain-specific databases

SQL dialect limitations

The evaluation uses SQLite.

The model should not be assumed to produce correct syntax for PostgreSQL, MySQL, SQL Server, Oracle, or other SQL dialects without additional testing.

SQL correctness

The model can generate SQL that is syntactically valid but semantically incorrect.

Generated SQL should be validated against the actual database schema and, where appropriate, tested before execution.

Model size

The base model contains approximately 3B parameters. While this makes the model relatively lightweight, it also limits its ability to handle highly complex SQL generation compared with substantially larger models.

Evaluation limitations

Execution accuracy was measured using schema-seeded in-memory SQLite databases with empty tables. Consequently, execution accuracy primarily measures structural and semantic SQL equivalence under the evaluation setup rather than correctness against realistic database contents.

Recommendations

For production applications:

• —Validate generated SQL before execution.
• —Prefer read-only database credentials.
• —Restrict permitted SQL operations.
• —Validate referenced tables and columns.
• —Use query timeouts and resource limits.
• —Test queries against representative data.
• —Do not expose unrestricted production database credentials to the model.
• —Consider SQL parsing and allowlisting before execution.

Training

Training Dataset

The model was fine-tuned on the sql-create-context dataset.

The dataset provides database schemas, natural-language questions, and corresponding SQL queries.

Fine-Tuning Method

The model was trained using QLoRA.

The base Qwen2.5-3B-Instruct model was loaded using 4-bit NF4 quantization with double quantization. The base weights were frozen while LoRA adapters were trained.

Training Configuration

The repository reports that LoRA rank 8 underfit during initial experiments, while rank 32 did not provide sufficient improvement to justify its additional adapter memory. Rank 16 was selected for the final training configuration.

Checkpoint Selection

The best validation checkpoint was reported at approximately step 200.

The best checkpoint was restored for the final model.

Evaluation

The model was evaluated on 7,858 held-out examples from the validation data.

The base-model comparison was performed on a 300-example subset using the repository's SQL extraction procedure.

Metrics

Exact Match

Exact Match compares normalized generated SQL with the normalized reference SQL.

normalise(prediction) == normalise(reference)

This is a strict metric. Semantically equivalent SQL queries can receive different scores if their textual formulation differs.

BLEU

BLEU measures token-level n-gram overlap between generated and reference SQL.

Execution Accuracy

Generated and reference SQL queries are executed against schema-seeded in-memory SQLite databases.

execution(prediction) == execution(reference)

This provides a measure of semantic equivalence that is less sensitive to different SQL formulations.

Results

• —Base-model measurements were obtained on a 300-example subset.

The repository reports a 13× improvement in exact-match accuracy compared with its measured base-model result.

Result Interpretation

The difference between exact-match accuracy and execution accuracy is important:

Exact Match Accuracy: 30.67%
Execution Accuracy:   55.03%

This indicates that some generated SQL queries differ from the reference query text while still producing equivalent results under the evaluation setup.

Therefore, exact match alone should not be considered a complete measure of SQL generation quality.

Base Model Comparison

The repository reports the following comparison:

Qwen2.5-3B-Instruct:
    Exact Match: 2.33%
    BLEU:        51.87%

Distill:
    Exact Match: 30.67%
    BLEU:        59.74%

The repository also notes that the base model can return SQL together with Markdown code blocks or explanatory text. SQL extraction methodology therefore has a significant effect on the measured baseline.

How to Use

This repository contains a PEFT adapter. The compatible Qwen2.5-3B-Instruct base model is required.

Installation

pip install torch transformers peft bitsandbytes accelerate

Load the Adapter

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)
from peft import PeftModel

base_model = "Qwen/Qwen2.5-3B-Instruct"
adapter_path = "path/to/adapter"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(base_model)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=quantization_config,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    model,
    adapter_path,
)

Generate SQL

schema = """
CREATE TABLE orders (
    id INTEGER,
    customer VARCHAR,
    amount FLOAT,
    status VARCHAR
);
"""

question = "What is the total revenue from completed orders?"

prompt = f"""
Schema:
{schema}

Question:
{question}

SQL:
"""

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=128,
)

response = tokenizer.decode(
    outputs[0],
    skip_special_tokens=True,
)

print(response)

Repository

The complete training and application code is available in the project repository:

https://github.com/MahboobAlam0/Distill---Fine-tuned-Qwen2.5-3B

The repository includes:

• —QLoRA training code
• —Evaluation scripts
• —Inference wrapper
• —FastAPI API
• —Streamlit interface
• —Kaggle training notebook
• —Training and validation data
• —Docker configuration
• —Tests

Serving

The project includes a FastAPI REST API and Streamlit interface.

FastAPI

uvicorn api.main:app --reload --port 8000

Example request:

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "schema": "CREATE TABLE products (id INTEGER, name VARCHAR, price FLOAT, stock INTEGER)",
    "question": "Which products are out of stock?"
  }'

Streamlit

streamlit run app/streamlit_app.py

Docker

docker-compose up

The repository exposes the API on port 8000 and the Streamlit interface on port 8501.

Reproducing Training

The repository includes a self-contained Kaggle notebook.

1. Upload kaggle/train_notebook.ipynb to Kaggle.
2. Select an NVIDIA T4 GPU.
3. Run the notebook.
4. Export the resulting LoRA adapter.

The repository reports approximately 2 hours for the documented training run on a single Kaggle T4 GPU.

Technical Specifications

Hardware

• —1 × NVIDIA T4
• —Approximately 15 GB VRAM

Software

• —Python 3.12
• —PyTorch
• —Transformers
• —PEFT
• —BitsAndBytes
• —TRL
• —NLTK
• —SQLite
• —FastAPI
• —Uvicorn
• —Streamlit
• —Docker

Adapter

• —LoRA rank: 16
• —LoRA alpha: 32
• —Approximately 21M trainable parameters
• —Approximately 114 MB adapter size

Environmental Impact

Training was performed on a single NVIDIA T4 GPU on Kaggle.

The project does not provide a measured carbon-emissions value.

License

The GitHub project is released under the MIT License.

However, this model is a fine-tuned adapter based on Qwen/Qwen2.5-3B-Instruct. The upstream Qwen model is listed under the Qwen Research License Agreement, so the upstream model's license and terms must also be respected.

For the complete upstream license terms, see:

https://huggingface.co/Qwen/Qwen2.5-3B-Instruct

For the project's MIT license, see the GitHub repository:

https://github.com/MahboobAlam0/Distill---Fine-tuned-Qwen2.5-3B

Citation

If you use this model or the associated project, please cite:

@software{mahboobalam_distill_qwen25_3b,
  author = {MahboobAlam0},
  title = {Distill - Fine-tuned Qwen2.5-3B},
  url = {https://github.com/MahboobAlam0/Distill---Fine-tuned-Qwen2.5-3B},
  license = {MIT}
}

Acknowledgements

This project builds upon:

• —Qwen2.5-3B-Instruct
• —Hugging Face Transformers
• —PEFT
• —TRL
• —BitsAndBytes
• —PyTorch
• —sql-create-context
• —SQLite

Model Card Contact

For questions, issues, or contributions, please use the GitHub repository's issue tracker.

Project: Distill — Fine-tuned Qwen2.5-3B Task: Natural language → SQL Base model: Qwen2.5-3B-Instruct Fine-tuning: QLoRA Training data: 28,000 examples Validation data: 7,858 examples Exact Match: 30.67% Execution Accuracy: 55.03% Adapter size: ~114 MB Hardware: 1 × NVIDIA T4