Why retrieval is not the same as learning
Large language models can draw on knowledge encoded in their parameters, retrieve passages from a database, and generate an answer from a supplied context. But unfamiliar problems often require a more active process: identifying what is missing, finding the right material, understanding a new principle, and using it in a solution.
In Incentivizing Self-Learning in Large Language Models via Reinforcement Learning, Lingkai Kong and colleagues study this capability under the name “self-learning.” Their central question is whether reinforcement learning can teach a model to study long-form reference material while solving a problem, rather than merely retrieve a preselected passage or consume a fixed context window.
The paper’s testbed is mathematical proof. Each exercise is paired with a structured textbook containing definitions, theorems, explanations, and proofs of general results. The solution to the exercise is withheld from the model. To succeed, the model must navigate the book, acquire relevant concepts, and construct a self-contained proof.
A textbook agent trained to study selectively
The authors build a dataset from 158 mathematics textbooks and extract 5,183 proof problems. The training collection contains 4,557 problems from 132 books. Evaluation separates held-out problems from books seen during training, unseen books in familiar subject areas, and books from an unseen subject area: probability and statistics.
The model interacts with a BookNavigator interface. It can inspect a table of contents, search section text by keyword, and read a complete section. This creates a loop in which reasoning and information gathering can influence one another. A failed derivation may reveal a missing definition; a newly read theorem may suggest a different proof strategy.
Simply rewarding the final answer, however, gives weak feedback about which retrieval decisions were useful. The researchers therefore modify Group Relative Policy Optimization with two additional signals.
- An exploration bonus rewards correct solutions that include successful textbook access. The bonus is stronger early in training and gradually decreases, while retaining a positive floor.
- Round-level tool credit evaluates individual textbook interactions as useful, neutral, or harmful. The resulting score redistributes learning credit across reasoning and tool-use rounds without changing the trajectory’s overall outcome signal.
A correctness judge evaluates the final proof against the hidden reference solution. A frozen copy of the initial model judges the quality of intermediate tool use. The authors’ goal is not to maximize the number of calls, but to encourage purposeful study.
Large gains on proof tasks
The headline result is a substantial reversal in the initial ranking. Before reinforcement learning, the self-learning protocol reached 51.5% overall proof accuracy, below the no-book condition at 55.7% and below fixed-retrieval baselines. After training, self-learning reached 79.5%, an improvement of 28.0 percentage points.
The trained self-learning policy exceeded the strongest fixed-retrieval baseline, Top-k Sections, by 6.4 percentage points overall. It also performed best on the held-out generalization splits, reaching 85.2% on unseen books in familiar subjects and 86.7% on the unseen-subject split. These results support the authors’ claim that adapting study to the evolving reasoning process can outperform placing retrieved material in the prompt ahead of generation.
Additional tests suggest that the improvement is not explained only by memorizing the paired textbooks or becoming better at solving problems without references. When textbook access was removed, the trained policy’s accuracy rose from 55.7% to 71.7%. With the paired textbook available, it reached 79.5%, leaving a 7.8-point benefit from reference access.
The policy also retained an advantage when the source section was removed or when the model received a different textbook from the same subject. The authors interpret this as evidence for a more general study strategy, although the experiments do not prove that the strategy is fully domain-independent.
Transfer beyond mathematics
The researchers evaluated the trained policy on BrowseComp-Plus, a deep-research-style retrieval benchmark, and CL-bench, which tests learning from supplied context. These evaluations used the benchmarks’ original protocols rather than BookNavigator.
Self-learning training improved the base model by 1.7 percentage points on BrowseComp-Plus and 2.3 points on CL-bench. The gains were selective: performance improved most in domain-knowledge reasoning and rule-system application, while all reinforcement-learning variants underperformed the base model on procedural task execution. The authors suggest that proof training may not sufficiently prepare a model to follow detailed operational workflows.
What the ablations reveal
The ablation results highlight a practical danger in outcome-only reinforcement learning. Correctness-only training raised proof accuracy, but the proportion of trajectories containing a tool call fell from 36.9% to 6.7%. A stronger solver had less immediate incentive to consult a book, even when exploration could matter for unfamiliar tasks.
Tool credit alone did little to restore exploration. The exploration bonus increased tool-using trajectories to 79.9%, and combining it with tool credit produced the strongest results: 79.5% overall, with the largest gains on unseen books and subjects. Importantly, the full method averaged 3.0 calls per trajectory, rather than simply maximizing tool use.
Limitations and practical implications
The evidence is promising but bounded. The main task is advanced mathematical proof, the reference collection is textbook-based, and correctness depends on an LLM judge using eight pessimistically aggregated reviews. Judge errors, evaluation prompts, and the quality of extracted textbook material could all influence the results. The paper also reports relatively modest transfer gains on external benchmarks, and the drop on procedural tasks shows that improved knowledge acquisition does not automatically produce better general agent behavior.
For developers, the broader lesson is that retrieval systems should be evaluated as learning processes, not only as answer-generation pipelines. Agents may benefit from deciding when to orient themselves, search, read deeply, verify assumptions, and stop. Training signals should reward useful information acquisition while penalizing redundant access, and should measure both final correctness and behavior on unfamiliar material.
The authors present self-learning as a potential way to use abundant long-form corpora as supervision without exposing exercise solutions. That possibility remains a research direction rather than a demonstrated route to autonomous model improvement. Still, the results show why “retrieve a passage” and “learn enough to solve the problem” should be treated as distinct capabilities.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.