Enterprise Text-to-SQL

Ask the fine-tuned model yourself

The QLoRA adapter — 70.86 % strict execution accuracy — answering in real time on a free GPU. Same weights, same prompt and the same 4-bit configuration every number below was measured with.

Open the live demo →

First question takes ~20 s while the model loads; after that it is a few seconds. Or explore all 453 recorded answers in the Explorer.

Base model

43.71 %

Fine-tuned

68.43 %

+ self-correction

70.86 %

Trainable params

0.917 %

Ablation — 453 held-out questions

Same questions, same harness, same database. Only the named component changes.

Where the gain actually came from

failure mode	base	fine-tuned
right rows, wrong columns	159	0
genuinely wrong rows	240	203

The benchmark's largest failure was returning correct data under a different column projection — "show orders in 2023" never says which columns to return. That went to zero. Genuinely wrong answers fell far less.

So most of the +40 points is the model learning this database's column conventions, not becoming dramatically better at SQL logic.

tier	n	base	fine-tuned	Δ
easy	126	0.0	23.0	+23.0
medium	63	52.4	52.4	+0.0
hard	144	3.5	84.7	+81.2
enterprise	84	13.1	52.4	+39.3

The hard tier — multi-table joins and subqueries — is where reasoning genuinely improved. medium did not move at all: a failure analysis traces 24 of its 25 remaining misses to one template whose gold renders months as EXTRACT(MONTH) while the model writes DATE_TRUNC — same 12 rows, different representation.

Two results that did not go as expected

Schema retrieval made things worse — twice

Showing only the retrieved tables cost the base model 1.55 points and the fine-tuned model 9.27. The adapter only ever saw full-schema prompts, so subsets are out of distribution for it — and the damage lands on hard (−20.8) and enterprise (−13.1), the tiers that need joins. Drop a table a join needs and the model invents one: hallucination nearly doubles.

With 12 tables the whole schema is 5,455 characters. Retrieval solves a problem this database does not have.

Self-correction: two rates, not one

repair success — now executes	11 / 19	57.9 %
repair correctness — right rows	5 / 19	26.3 %

Six of the eleven "successes" turned a crash into a confident wrong answer. One rate would have claimed 57.9 % and hidden that.

Repair works when the error names the fix (column "method" does not exist → rename to payment_method) and fails when it names only the symptom (ambiguous column reference).

The pipeline

question → prompt (frozen template) → Qwen3-8B + QLoRA adapter → SQL
        → static validation   parseable? one statement? read-only? real tables?
        → PostgreSQL          read-only session, 30 s statement timeout
        → success → rows
        → failure → repair    the database's own error fed back, retried once

How correctness is judged

By execution, never string similarity. Every prediction runs against PostgreSQL and its result set is fingerprinted. These are different strings and both count as correct:

SELECT COUNT(*)           FROM customers WHERE country = 'India';
SELECT COUNT(customer_id) FROM customers WHERE country = 'India';

Order is compared only where the gold query has an ORDER BY. A second, projection-tolerant metric wraps the prediction in a subquery and asks for the gold columns — that is what separates "wrong columns" from "wrong rows", and it is the whole reason the 159 → 0 finding above is visible at all.

Guardrails

Leakage-free splits by template group	74 train / 48 test templates, zero overlap
Prompt fingerprint asserted at every stage	`8288e41a496531a9`
Schema fingerprint asserted at every stage	`d03619e711661bc5`
The GPU host that generates never receives gold SQL	asserted both ends
Oracle self-test — gold SQL through the scorer	must return 100 %

Training

base	`Qwen/Qwen3-8B` @ `b968826d9c46`, 4-bit NF4
adapter	LoRA r=16, α=32, all 7 attention + MLP projections
trainable	43,646,976 of 4,761,498,624 — 0.917 %
hardware / time	one free Kaggle T4, 1 epoch, 7 h 05 m
loss	completion-only — prompt masked to −100
supervised token share	2.41 %
train / eval loss	0.083 / 0.145 (iteration 1: 0.006 / 0.335 — memorised)

Why completion-only loss is the detail that matters. Each example is ~1,490 prompt tokens and ~46 completion tokens, and the prompt is ~97 % schema, byte-identical across all 2,133 examples. Training on the full sequence sends ~97 % of the gradient into memorising a schema the model is handed at inference — and the loss curve looks excellent throughout, because predicting a constant is easy.

Limitations

Schema-specific. It learned this database's conventions — that is most of the measured gain.
Roughly half its answers are still wrong under strict scoring. Classifying all 217 failures: 118 return the right rows under a different column set (the question never said which columns), 91 are real logic errors — mostly business definitions the prompt never states — and 8 are detectable errors. Row-level accuracy is 78 %; the strict number stays the headline.
Test questions are template-generated, from the same generator as training. Real user phrasing — abbreviations, typos, ambiguity — is untested. The biggest caveat on the headline number.
One epoch, one seed, one run. No variance estimate.
Silent wrong answers are the real risk. SQL that runs and returns the wrong rows is indistinguishable from a correct answer without a gold reference — and repair cannot help, because there is no error to feed back.