Sketchpad: certified exact arithmetic with a small vision-language model
Language models can produce impressive answers to long-addition and multiplication tasks, but benchmark accuracy is not a proof of correctness. A model may fail on an unseen input, and the failure can look like a finished, plausible answer. External calculators provide exactness, but they move the arithmetic outside the model.
In Sketchpad: certified exact arithmetic with a small vision-language model, Mostafa ‘Neo’ Mohsenvand proposes a different division of labor. A small vision-language model performs local arithmetic steps on a rendered worksheet, while a conventional runtime controls the procedure and moves a bounded window across the sheet. The authors’ central claim is that this structure turns an unbounded arithmetic problem into a finite certification problem.
Why a bounded view matters
Each model call sees only three cells, an operation token, and output placeholders. It emits a digit or intermediate value, a carry or borrow, and the next operation. The runtime writes those outputs to the sheet and advances the window. No hidden reasoning trace, earlier answer, or other state is passed from one step to the next.
That restriction is important. Under the paper’s formal model, every possible input to a step belongs to a finite set of views. If the model produces the correct output for every view, induction shows that the complete procedure is correct for runs of any length. The authors formalized this argument in Lean.
The paper also proves a converse-style limitation: if a step can read the entire history of earlier steps, as in ordinary chain-of-thought decoding, no finite test suite can guarantee correctness on all longer runs. A decoder can be constructed that passes every finite collection of tests and then fails on a longer context. The result does not say that chain of thought always fails; it says that finite testing alone cannot provide the same universal guarantee.
How Sketchpad is trained and checked
Sketchpad adapts Qwen3.5-0.8B with a low-rank adapter trained directly on exhaustive local-operation tables. The tables cover column addition, long multiplication, subtraction, quotient estimation, and decrementing an estimate. Together they contain 2,446 views.
The model is trained not only to select the correct output token, but also to maintain a positive logit margin. The deployed certificate evaluates every view on the target execution path: an Apple M4 laptop GPU running the model in a browser through WebGPU and 16-bit arithmetic. All 2,446 views passed. The smallest reported margin was 3.02 logits, which the authors say provides tolerance for bounded numerical changes below half that margin.
This is stronger than checking a random sample of large arithmetic problems, but it is not a proof that every part of the system is correct. The theorem depends on the runtime presenting only valid table views and implementing the reference algorithms correctly. The paper states that these runtime obligations were asserted and tested, but the concrete layouts and procedures were not fully formalized in Lean.
What the experiments showed
The certified procedures cover signed addition and subtraction, long multiplication and division, square roots, and greatest common divisors at arbitrary input length, subject to the runtime and representation assumptions. In a pre-registered browser test, Sketchpad solved 200 of 200 problems exactly: 50 each for division, square roots, greatest common divisors, and signed sums or differences. Median times ranged from 2.0 seconds for signed arithmetic to 9.8 seconds for square roots on the reported laptop setup. The slowest problem took 16.5 seconds.
In a separate comparison involving 220 problems, the system solved all 220 sums and products. The strongest of ten other tested models, GPT-6.1 Sol, solved 217. The paper reports that the alternative models’ errors included silent mistakes on long sums, while several open models also exhausted their output budgets. These results demonstrate the benefit of the worksheet design on this test, not a general ranking of mathematical ability.
The authors further composed the certified arithmetic operations into a symbolic runtime for polynomial expansion, equation solving, and factorization. The runtime checks roots and some derived expressions, but those checks are not complete proofs that every root or coefficient was found. The demonstrated algebra system is also limited: the paper describes low-degree polynomial tasks and linear systems of up to three equations.
Preserving general language ability
Fine-tuning an arithmetic adapter initially damaged instruction-following performance, illustrating a familiar catastrophic-forgetting problem. Sketchpad addresses this with a scalar gate on the adapter. When the gate is zero, the adapter branch contributes an exact floating-point zero, and the authors report bit-for-bit recovery of the base model in their tests. When the gate is active, the model executes the sheet procedures.
A dialogue adapter was trained to decide when to invoke the sheet. On held-out invocation tasks, the paper reports precision of 0.994 and recall of 0.989. However, the broader result is mixed: on fresh GSM8K problems, the dialogue adapter scored 5.8 points below the base model. The authors attribute this loss to multi-step reasoning rather than arithmetic execution. This distinction matters: certified arithmetic does not automatically certify the model’s interpretation of a word problem or its decision to call the tool.
Practical implications and open constraints
Sketchpad’s most useful idea is architectural. If an AI system must perform a narrow, high-stakes computation, giving a model unrestricted scratch space may make universal guarantees impossible. A bounded external memory, exhaustive local certification, and a conventional controller can instead make the model’s step-level behavior auditable.
That approach could be relevant to browser-based calculators, educational software, or agent workflows that need exact numerical subroutines without sending every calculation to an external service. It also suggests a practical certificate format tied to model-file hashes, runtime versions, device precision, and glyph definitions.
There are clear trade-offs. The model requires a specialized rendered input format, each step takes roughly a tenth of a second on the reported laptop GPU, and the guarantee applies only to the certified execution path. New hardware may require renewed checking. Parsing, transcription, runtime control, and natural-language reasoning remain outside the arithmetic certificate. The authors therefore demonstrate a credible pattern for certifying a useful neural component—not a general solution to reliable mathematical reasoning.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.