The Lexical Imprint of Pre-training Data
Understanding what shapes the zero-shot performance of large language models (LLMs) remains a central question in AI research. A recent study investigates how the lexical characteristics of a model’s pre-training corpus influence its behavior on downstream benchmarks. The authors introduce a simple, model-free metric—word-level unigram cross-entropy (UCE)—to quantify the alignment between the lexical distributions of pre-training data and evaluation benchmarks. Their findings reveal that this lexical alignment consistently predicts zero-shot accuracy, offering new insights into the role of training data in shaping LLM capabilities.
Problem: How Does Pre-training Data Influence Zero-Shot Performance?
Modern LLMs are typically evaluated on zero-shot benchmarks, where models must perform tasks without task-specific fine-tuning. However, the relationship between the pre-training corpus and benchmark performance is complex and often entangled with factors like model architecture, scale, and training recipe. Prior work has observed correlations between term frequencies in pre-training data and few-shot or zero-shot performance, but these analyses often rely on pretrained checkpoints, making it difficult to isolate the effect of corpus identity alone.
The authors ask a fundamental question: To what extent does the lexical alignment between the pre-training corpus and the benchmark target explain zero-shot accuracy? They approach this by measuring how closely the word distributions of the pre-training data and benchmark match, hypothesizing that better lexical alignment leads to improved zero-shot performance.
Method: Measuring Lexical Alignment via Unigram Cross-Entropy
Directly estimating conditional distributions at corpus scale is infeasible due to data sparsity and computational constraints. Instead, the authors propose using word-level unigram cross-entropy (UCE) as a proxy for lexical alignment. UCE measures the cross-entropy between the empirical word-frequency distributions of the pre-training corpus (reference) and the benchmark dataset (target). Lower UCE indicates closer alignment.
This metric is tokenizer-agnostic and model-free, relying solely on normalized word counts after Unicode normalization, punctuation stripping, and whitespace tokenization (without lowercasing). The authors apply smoothing to handle words present in the benchmark but absent in the corpus.
They test UCE at three levels:
- Corpus level: Comparing entire pre-training corpora against benchmarks.
- Document level: Selecting training documents to minimize UCE and thus increase lexical alignment.
- Prompt level: Routing prompts to specialized drafter models whose training corpora best align lexically with the prompt.
Evidence and Results
Corpus-Level Analysis
The authors conduct controlled from-scratch pre-training experiments with decoder-only Transformer models at three scales (400M, 1.33B, 3.36B parameters) on four diverse corpora: FineWeb-Edu and DCLM (high-quality), C4 (high-quality but older), and OpenWebText (lower quality). They evaluate zero-shot accuracy on 10 benchmarks spanning commonsense reasoning, science, and general knowledge.
Across all benchmarks and scales, lower UCE between the pre-training corpus and the benchmark correlates strongly with higher zero-shot accuracy. For example, FineWeb-Edu and DCLM, which have lower UCE values, consistently outperform C4 and OpenWebText. This relationship holds robustly across multiple random subsets of the corpora, indicating stability beyond specific data samples.
Document-Level Data Selection
To move beyond correlation, the authors develop the UCE-filter, a data selection method that ranks and retains training documents that reduce corpus–benchmark UCE. By selecting the 20% most lexically aligned documents from DCLM and FineWeb-Edu, they train models from scratch on a fixed token budget and observe significant zero-shot accuracy gains on the targeted benchmarks compared to random sampling and a strong importance-resampling baseline (DSIR).
Importantly, these gains concentrate on the benchmarks used to define the target distribution for selection, with little to no improvement on out-of-target tasks. This specificity supports the interpretation that lexical alignment drives the observed accuracy improvements rather than generic data quality effects.
A large-scale case study on a 30B-parameter model further demonstrates that replacing the web component of a production pre-training mixture with its UCE-filtered subset preserves most of the benchmark gains at roughly 10× fewer tokens, highlighting the token efficiency of alignment-focused data selection.
Prompt-Level Routing for Efficient Inference
At inference time, the authors apply UCE to route each prompt to the drafter model whose training corpus is most lexically aligned with the prompt’s word distribution. Using a pool of specialist drafters trained on subdomains (math, code, science) and a generalist drafter baseline, they show that UCE-based routing improves throughput and decoding speed across diverse benchmarks, including those outside the specialists’ domains.
Limitations and Considerations
The authors carefully discuss the scope and limitations of their approach:
- Lexical granularity: UCE captures only unigram word distributions and ignores higher-order conditional structure such as syntax, multi-token composition, and factual co-occurrence. It does not predict performance on grammar, arithmetic, or non-English benchmarks.
- Tokenization assumptions: The method relies on whitespace tokenization, which may not generalize well to non-segmented scripts like Chinese or agglutinative languages.
- Target specificity: The UCE-filter’s gains are upper bounds under a known target distribution estimated from benchmark data, limiting claims about zero-shot generalization to unseen tasks.
- Benchmark contamination: The authors audit and remove overlapping benchmark items and training documents to mitigate contamination effects, concluding that contamination does not explain the lexical alignment–accuracy relationship.
- Computational cost: While the UCE-filter is efficient and scalable, the paper reports substantial compute requirements for from-scratch training and large-scale data selection.
Practical Implications
This work offers a straightforward, interpretable metric to quantify how closely pre-training data matches downstream evaluation targets at the lexical level. Practitioners can leverage UCE to:
- Diagnose and interpret zero-shot model performance in relation to training data.
- Guide data curation and selection to improve task-specific accuracy efficiently.
- Implement prompt-level routing strategies to accelerate inference by leveraging specialized models.
Moreover, the findings caution against over-reliance on benchmark-aligned data, which may inflate scores without improving generalization. Reporting UCE alongside accuracy can provide transparency and help contextualize model capabilities.
Conclusion
The authors demonstrate that pre-training corpora leave a measurable lexical imprint on zero-shot LLM behavior, captured by word-level unigram cross-entropy. This lexical alignment not only correlates with accuracy across benchmarks but also enables targeted data selection and inference-time routing to improve efficiency and performance. While UCE does not capture all aspects of language understanding, it offers a valuable, scalable tool for probing and controlling the influence of pre-training data on model behavior.
“Pre-training pulls a model’s predictive distribution toward its corpus’s empirical conditional patterns, leaving a measurable lexical imprint on zero-shot behavior.”
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.