Why text-to-SQL needs memory
MEM-SQL: Memory-Empowered Text-to-SQL starts from a practical observation: people often ask several questions about the same database. Yet many text-to-SQL benchmarks treat every question as an isolated task. That makes it difficult to measure whether a system can reuse a verified join, calculation, filter pattern, or schema concept from an earlier request.
The authors introduce two related contributions. HARE, the History-AwaRe Text-to-SQL BEnchmark, evaluates how systems adapt historical SQL while tracking both accuracy and generation cost. MEM-SQL is the proposed memory architecture for doing that adaptation. Together, they frame text-to-SQL as an incremental database interaction problem rather than a sequence of unrelated prompts.
HARE tests different kinds of reuse
HARE contains 1,600 requests derived from 823 verified queries across Spider, Spider 2.0, BIRD, and BEAVER. Its annotations distinguish what a new request preserves from what it changes. The benchmark includes rephrasing and parameter changes, but also harder cases involving reusable intermediate computations, shared join paths, and shared schema concepts. Six composite categories require evidence from more than one historical source.
This design matters because an executable query is not automatically a valid adaptation. HARE’s construction and validation process checks the question’s meaning, execution, and whether the requested relationship to historical queries is actually preserved. For example, a request might retain a join path while changing the grouping or calculation performed over the joined rows.
MEM-SQL’s three-layer memory
MEM-SQL stores verified history at three granularities:
- Templates: parameterized whole-query structures for requests that largely preserve the original query.
- Components: reusable SELECT blocks, CTEs, subqueries, and related intermediate computations.
- Schema concepts: mappings between domain language and tables, columns, measures, and join relationships.
Each memory item pairs a searchable description with a structured payload. The description supports retrieval, while the payload preserves SQL structure or schema mappings. This separation lets the system reuse exact computational patterns without treating every historical query as an indivisible demonstration.
The central control mechanism is a coverage gate. MEM-SQL decomposes a question into layer-specific retrieval requirements and checks each requirement separately. A layer is accepted only when every requirement assigned to it has a sufficiently strong match. A highly similar match for one part cannot conceal missing evidence for another part.
The system also uses On-demand Schema Linking, or OSL. It first tries to generate SQL from qualifying memory evidence. Schema linking is invoked only when no layer qualifies or when the generator judges the retrieved evidence insufficient. The resulting SQL is checked with a database dry run and can be repaired when validation fails.
Reported results: better accuracy at lower cost
Across six language models, the authors report that MEM-SQL raises mean execution accuracy from 37.1% for direct generation without history to 50.0%. They also report 76.6% fewer input tokens and 68.8% lower latency on average. MEM-SQL outperformed the three tested memory baselines on every backbone in the main comparison.
The gains are not uniform. Memory helps most when a new request preserves an entire query structure or a substantial join path. Rephrasing and ordinary parameter changes benefit particularly strongly because the system can reuse verified computation instead of reconstructing it from schema descriptions. The authors also report gains on join-heavy BEAVER requests and on parameter changes in BIRD.
The ablations clarify the roles of the memory layers. Removing the template layer reduced execution accuracy by an average of 5.3 percentage points and increased token use across the evaluated models. Removing the component layer hurt composite requests more than single-category requests, supporting the case for storing intermediate computations separately. Schema concepts contributed more consistently to efficiency than to accuracy: removing them changed accuracy little but generally increased token use or latency.
Requirement-specific retrieval also appears useful. On a 240-question calibration set, retrieval atoms reached 94.58% micro recall while returning fewer items than flat retrieval. The authors report that this approach improved execution accuracy over full-question retrieval on all six backbones while keeping the evidence bundle smaller.
The hard case is composition
The most important caution is that retrieving multiple relevant queries does not mean a model can combine them correctly. Accuracy fell from 56.7% for requests reusing one verified query to 29.7% for requests requiring several sources. The paper identifies new query structures and combinations involving shared joins as particular bottlenecks.
This is not simply a retrieval problem. The authors report that MEM-SQL often retrieved all required sources for difficult multi-source requests, yet accuracy remained low. The model still had to place partial computations into a new structure, preserve correct aggregation scope, and avoid changing the meaning of joins. Aggregation and grouping errors remained the largest class of memory-route failures, while table-selection errors decreased relative to a general-purpose memory baseline.
Practical implications and limits
For a production analytics assistant, the paper suggests a concrete design principle: retain verified SQL at multiple levels rather than storing only full prompt-and-query examples. Whole-query templates can handle routine variations, reusable components can support composition, and schema concepts can preserve domain terminology and relationships. A coverage check should be explicit, and schema retrieval should be deferred when historical evidence is genuinely sufficient.
However, the results do not show that memory eliminates the need for schema access or human oversight. OSL consistently reduced input tokens and latency when added to other memory designs, but its accuracy benefit depended on whether those designs retrieved complete evidence. MEM-SQL also produced more invalid SQL than one baseline on some memory-only routes, and aggregation errors remained substantial.
The authors identify three important limitations. HARE does not model schema drift, such as changing definitions or database structures. MEM-SQL is evaluated with static memory, leaving the costs of adding, updating, and retiring memories open. Finally, execution accuracy can reject harmless formatting differences and can accept a query that returns the right result for the wrong reason. The paper therefore points toward intent-sensitive evaluation as a necessary next step.
Overall, MEM-SQL demonstrates that verified query history can make text-to-SQL both more accurate and cheaper, but only when the retained evidence matches the new request. The strongest opportunity is routine adaptation; the largest unresolved challenge is composing several partial memories into a correct new query.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.