Glance: The Generator Is Its Own Reranker

Glance: The Generator Is Its Own Reranker

Retrieval-augmented generation usually divides evidence selection and answer writing between two models. A cross-encoder or another language model scores candidate passages, then a generator reads the winners and produces the response. The authors of Glance: The Generator Is Its Own Reranker ask whether that extra reranker can be removed: if the generator must read the passages anyway, can it also judge them using computation that is already being performed?

Glance is an experimental answer to that question. It gives a generator a lightweight decision adapter and a pointer head that scores passages from a shared prefill. The paper reports promising retrieval and downstream question-answering results, while also making clear that the system is still a proof of concept rather than a drop-in replacement for every production reranker.

How Glance shares computation

In a conventional pipeline, the reranker processes each query–passage pair, often in separate model calls, before the generator processes the selected context. Glance instead places candidate passages into one shared context. It then branches into short, passage-specific relevance probes. Each probe produces a hidden representation, and a 256-dimensional pointer head converts those representations into relevance scores.

The backbone is a frozen 9B model, while a rank-16 LoRA adapter and the pointer head provide the decision capability. The same resident backbone can operate in base chat mode, general decision mode, or reranking mode. The authors report that each additional decision skill requires approximately 0.1 GiB for its adapter and head, although total inference memory also depends heavily on candidate count and cache handling.

The main Glance pipeline prefills all candidates, selects the top passages, and then prefills those passages again with the decision adapter disabled before generating an answer. A second variant, Glance-Activated, applies the adapter only to the probe tokens. That allows the base model’s original passage cache to be reused for generation, although the continued cache still contains all candidate passages rather than only the selected ones.

What the experiments found

The authors evaluated Glance on SciFact and NFCorpus, using BM25 to retrieve the initial candidate set and nDCG@10 to measure ranking quality. With no retrieval-specific training, the shared packed approach reached 0.715 on SciFact and 0.341 on NFCorpus. The bge-reranker-v2-m3 cross-encoder scored 0.721 and 0.326 respectively.

After 39 minutes of fine-tuning on 4,000 MS MARCO queries, the packed Glance configuration reached 0.741 on SciFact and 0.345 on NFCorpus. Those results exceeded the reported cross-encoder scores by 0.020 and 0.019, though the paper says the BEIR differences were not significance-tested. The training data did not include either BEIR or HotpotQA.

Sharing the prefill also reduced the cost compared with scoring every candidate through a separate call to the same 9B decision model. In the zero-shot experiment, Glance was about 3.6 times faster than sequential decision calls on SciFact and 3.7 times faster on NFCorpus. However, the dedicated cross-encoder remained substantially faster, with reported query latencies of 31 to 60 milliseconds compared with hundreds of milliseconds for the shared Glance runs.

The scaling results are particularly notable but should be interpreted carefully. On a 100-query SciFact subset with shorter passages, the trained-long adapter improved as the candidate pool grew, reaching 0.717 nDCG@10 at 200 candidates, compared with 0.659 for the cross-encoder in that experiment. That 200-candidate run involved 43.4k context tokens and took about 15.8 seconds per query. Current cache branches also caused memory to grow with candidate count, reaching 56.4 GiB at 100 candidates in the reported setup.

Does better ranking improve answers?

On 300 HotpotQA distractor questions, the authors compared answers generated from the top three passages selected by Glance and by the cross-encoder. With the trained adapter used for reranking and the base generator used for answering, Glance improved answer F1 from 0.684 to 0.729. The paper reports a paired-bootstrap 95% confidence interval of 0.014 to 0.076 for that difference.

This result is more useful than ranking scores alone because it tests whether the selected evidence helps the same generator answer. Interestingly, Glance did not always retrieve more gold paragraphs in its zero-shot configuration. The authors suggest that it may select evidence the generator can use more effectively, but they explicitly caution that the training labels describe passage relevance, not answer utility. The experiments therefore demonstrate improved answers in the tested setting, not that Glance has learned to optimize answer quality directly.

The activated variant offers a different systems benefit. In a smaller, single-seed evaluation, generation continued from the original passage cache and reduced median answer latency from 181 milliseconds to 138 milliseconds. The authors report similar ranking quality to a matched control, but the generator still saw all passages. This is cache reuse, not yet a full implementation of selecting and removing losing passages without another read.

Limitations and practical implications

The evidence has several important boundaries. Evaluation covers two retrieval datasets, one QA dataset, and one hybrid 9B backbone family. Dataset sizes are in the hundreds, many latency measurements used different GPUs and partially shared environments, and the serving path was not fully optimized. The cross-encoder remained faster per query in the reported tests.

Glance’s scores can depend on passage order because every probe sees the shared context. The authors found that average order effects were small but not negligible for individual passages. Zero-shot quality also degraded beyond 20 candidates in one study, while batching multiple queries reduced throughput in the current implementation. The proposed remedy—letting branches share a cache without copying it—remains future work.

For practitioners, Glance is most compelling when model residency, cache reuse, and unified serving matter more than minimum per-query latency. A RAG system could keep one generator resident while attaching small decision skills for reranking and other structured choices. Teams should nevertheless benchmark it against a small cross-encoder on their own hardware, measure memory at realistic candidate counts, test passage-order sensitivity, and evaluate answer quality rather than retrieval metrics alone.

The paper’s central contribution is therefore architectural: it shows that a generator can perform passage-specific reranking from a shared prefill. The results are encouraging, especially for tightly coupled RAG systems, but broader backbones, larger evaluations, optimized kernels, and stronger latency comparisons are still needed before the approach can be considered a general replacement.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close