I built a text-to-SQL system over raw Retrosheet game logs (2007 to 2023). Before the model writes SQL, it gets a profile of what each column really holds, and names, teams and dates are looked up in the database.
On 100 questions with verified answers, Claude Haiku 5.5 went from 74 to 91 correct. That is level with Sonnet 5.5 asked directly (86), at about a tenth of the estimated cost. It grew out of my MSc AI dissertation at the University of Bath (2024).
Columns 1 to 5 are the question tiers. C is correct of 100, $ is the estimated cost per 1,000 questions, FIX is questions the pipeline turned from wrong to right.
My dissertation, Natural Language to SQL: A Comparative Study of Strong and Weak Supervision Models for Database Querying, asked how close we are to letting anyone question a relational database in plain English.
I compared the two main families of models. Strongly supervised models learn from annotated SQL and lead the WikiSQL benchmark, with SeaD at 93.0% execution accuracy. Weakly supervised models learn from questions and answers alone, with TAPEX the strongest at 89.5%.
I analysed the 72,233 examples in WikiSQL and found the queries behind them are mostly simple lookups on a single table. Strong benchmark scores were being earned on questions far easier than the ones people ask of real data.
I ran TAPEX, a weakly supervised neural SQL executor, on a baseball database I built from Retrosheet game logs and Toronto Blue Jays team statistics, and annotated question sets by hand to see where it held up.
TAPEX answered direct lookups well, such as how many doubles a team hit in a season. It struggled once a question needed several steps, a comparison across rows, or arithmetic across columns. The gap was not raw model skill: success on real data depends on understanding how that data is stored, and the benchmarks did not measure it.
Hybrid systems that combine learned models with structure, and evaluation broken down by query complexity and domain. This project builds both: a language model writing SQL with the data facts supplied by code, measured on 100 verified questions across five tiers of difficulty.
Each question runs the bases. Only third base calls a model; everything else is ordinary code that gives the model better facts going in and checks the query coming out. If the query fails or returns nothing, the runner goes back to third and the model gets another try, up to two more. Press play to follow a real recorded run of question q014.
The "without it" plates on the link and profile stations show Haiku's score with that stage removed (91 with every stage). Every model call goes through a record-and-replay harness, so these runs are recordings replayed exactly.
'U' for upheld is a sensible guess. The column never holds it, so the query ran without error and returned no rows. Nothing in that result told the model what was wrong.
The two queries differ only in the protest code. The model was not short of SQL skill. It was short of one fact about this data: which codes the protest column uses. The profile line listed the only three values present, so the model chose from those instead of inventing one. The linker supplied SFN for the Giants.
Questions like this, where a stored code has to be decoded, make up 37 of the 100. Haiku answered 23 of them directly and 32 with the pipeline.
Each tile is one real question from the test set. Scroll on and the board is reset for each stage of the story. Hover or tap a tile to read its question; click it to replay that question below.
I wrote 100 questions about the game logs, in five tiers of 20, each with a verified answer. Rows on the board are the tiers.
Questions mix skills: decoding stored codes, home or away, data traps, fan vocabulary and joins on player IDs.
First I gave Claude Haiku 5.5 each question and the table definitions, nothing more. Cream tiles are right answers: 74.
Most misses fell in two tiers: 8 wrong in tier 1 and 10 wrong in tier 4.
Then the same model through the pipeline: names linked to database codes, a profile of what each column holds, and a checked, sandboxed run. It answered 91.
Lit tiles are questions it now gets right; the red tile is the one it lost. Tier 4 went from 10 to 17.
Regrouped by what changed on the same questions: the pipeline fixed 18 and broke 1. 73 were right both ways and 8 wrong both ways.
A McNemar test on that 18 to 1 split gives p = 0.0001, so the gain is very unlikely to be chance.
My dissertation found that a table-reading model (TAPEX) failed on multi-step questions and indirect schemas, and proposed evaluating by complexity. Here, Haiku's direct errors track the tier only weakly. They track whether a stored code must be decoded much more closely.
Code decoding: 23 of 37 directly, 32 of 37 with the pipeline.
Run Haiku both ways and send only the 25 questions where the two answers disagree (ringed) to Opus. Replayed on the recorded runs, that scores 95 at about $1.37 per 1,000 questions, and no right answer turned wrong.
This is a simulation on recorded runs, not a separate live run.
Hover a tile to read the question. Click to replay it.
Four Claude models, each asked directly and through the pipeline, on the same 100 questions. Each light is one question in test order, lit when answered correctly (two rows of 50). The cream bar is the estimated cost per 1,000 questions on a shared scale. Fable 5.1 scored the same as Opus 5.5, 97 asked directly and 99 with the pipeline, at about two and a half times the estimated cost.
Three mismatches between the wording of a question and the values stored in the database account for most of what the pipeline fixed. Each card shows a real question. The struck-through value is what the model wrote without access to the data.
I removed one stage at a time from Haiku's pipeline and reran the 100 questions. Each variant ran once, so these figures are directional. Tiles are the questions that changed; click one to replay it.
Each arrow connects a model asked directly to the same model with the pipeline. Accuracy rises substantially and cost rises slightly. The two small grey dots are Haiku's pipeline with one stage taken out: without linking names to database values it scored 83, and without the column profile it scored 86. The hollow amber diamond is a simulated cascade, in which Haiku runs both ways and a question goes to Opus only when the two answers disagree. Replayed on the recorded runs, it escalates 25 of 100 questions and scores 95, matching Sonnet's pipeline at about 70% of its cost.
Pick any of the 100 questions and any model, then step through link, profile, write, check and answer, with the pipeline and asked directly. Both rows are replays of recorded runs.
The system. A Python package with 38 tests. DuckDB opens the data read-only with external access disabled; sqlglot allows a single SELECT and blocks writes and file functions; a timeout and a row cap bound each query. Name, team, park and date aliases are built from the database itself. A ledger records each model call's stage, tokens, cost and latency.
How the runs were made. Every model call goes through a record-and-replay harness. The pipeline writes out each prompt it needs, keyed by a hash of its contents, and each model tier answers the batch. Replies are stored next to their prompts, so any run can be re-scored, inspected or replayed offline with the same pipeline, checks and database a live deployment uses; the replays on this page are those recordings. Each run is scored once, so comparisons rest on paired counts (18 fixed and 1 broken for Haiku). Costs are estimated from text length at list prices, so the ratios between runs are the figures to compare.
Related work. This is a lightweight, single-model variant of modular pipelines such as DIN-SQL and CHESS. As in CHESS, grounding the question's values in the database mattered much more than repair. An agent mode, in which the model chooses its own exploratory queries, is implemented and tested; comparing it with the pipeline is the next experiment.