DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction
Adding layers to a Transformer should increase its representational capacity, but deeper is not automatically better. Qin and colleagues argue that Pre-Layer-Normalization Transformers can suffer from depth-wise information dilution: early features are repeatedly folded into a residual stream, making them harder for later layers to retrieve. Cross-layer connections can help, but many designs add computation, memory traffic, or system complexity.
The authors revisit a more targeted mechanism: sharing the attention key-value (KV) states produced by some layers with other layers. Their central claim is that KV sharing is more than an inference optimization. Because a deep layer consumes a source layer’s KV representation, its gradients also flow back through that representation during training. The resulting connection can act as a structural regularizer for the geometry of keys and values.
From cache compression to gradient regularization
In ordinary attention, every layer computes and stores its own keys and values. During autoregressive decoding, those tensors form the KV cache, which can consume substantial memory. Cross-layer sharing reduces that burden by allowing multiple layers to use the same or reconstructed KV states.
Qin et al. focus on what this reuse does during optimization. They distinguish the roles of keys and values. Keys determine retrieval through softmax competition, so shared training signals can reinforce a smaller set of discriminative retrieval directions. Values are read through the attention pattern as a content channel; different layers can provide complementary gradients that encourage a richer value subspace. The paper measures these changes using effective rank, a summary of how broadly activation energy is distributed across feature directions.
This explanation is conditional rather than a universal theorem. The paper’s theoretical analysis assumes particular alignment patterns in the update covariance. Its experiments are intended to test whether those assumptions and predicted trends appear in practice.
How DepthGraft works
DepthGraft divides the Transformer into approximately equal shallow, middle, and deep stages. The shallow and middle stages compute KV normally. Each deep layer then reconstructs its KV from two earlier sources: a stride-aligned shallow layer and a corresponding middle layer. Learnable, channel-wise coefficients determine how much each source contributes to the reconstructed key and value.
This topology is designed to balance two competing concerns. The source and reuse layers are far enough apart to expose different query spaces, creating more useful cross-layer gradient interactions than adjacent-layer sharing. At the same time, the reuse edges are distributed across many source layers rather than concentrated on one or two global anchors. The two sources also provide memory bases at different depths, which the authors associate with lower-level and more abstract information.
The deep stage does not materialize independent KV tensors. As a result, the authors report roughly one-third less KV-cache storage. During prompt processing, all KV tensors needed for later decoding are available after the shallow and middle stages, allowing the deep stage to avoid processing the full prompt for cache construction. The design is also intended to fit pipeline and virtual pipeline parallelism when layers are placed appropriately.
What the experiments show
The authors train dense models from 0.5B to 1.5B parameters and DeepSeekMoE-style sparse models from 3B to 16B total parameters. They compare against matched GQA or MLA baselines and evaluate knowledge, reasoning, code, and Chinese-language benchmarks.
DepthGraft improves average benchmark scores at every reported dense scale: by 0.75 points at 0.5B, 0.85 at 1B, and 2.42 at 1.5B. On the 16B MoE GQA model, the reported average rises from 43.43 to 47.04. Individual gains include 7.00 percentage points on MMLU-Pro, 6.61 on CMMLU, and 6.41 on C3. The method also improves the 9B model under both GQA and MLA, suggesting that the result is not tied to one attention implementation.
Ablations support the proposed mechanism. A version that reuses KV but detaches the backward path performs close to the baseline, whereas enabling cross-layer gradients produces a larger improvement. The full two-source hierarchical design outperforms single-source alternatives and the tested two-stage and four-stage layouts. In comparisons on the 16B MoE model, DepthGraft also beats adjacent sharing, global-anchor sharing, and FusedKV under the paper’s matched protocol.
The efficiency results are similarly practical. On the reported 9B MLA and 16B GQA systems, time to first token falls increasingly as context grows, reaching reductions of 26.8% and 28.8%, respectively, at 128K context. Time per output token changes by less than 1% across the tested context lengths. The paper also reports higher RULER averages after long-context extension and retention of the pretraining advantage after supervised fine-tuning.
Limitations and implementation questions
These findings are promising, but they do not establish that DepthGraft is universally superior. The experiments use a limited set of model families, training recipes, and primarily private pretraining data. The largest reported model is 16B total parameters, and the paper does not demonstrate results at the scale of frontier production systems. Reported latency depends on hardware, kernels, placement, and runtime integration; the authors note that specialized kernel work may be needed to approach the theoretical savings.
The effective-rank story is also an interpretation supported by diagnostics, not a complete account of why downstream scores improve. The theory relies on explicit spectral and gradient-alignment assumptions, and the authors identify more rigorous analysis as future work. Finally, the method constrains deep-layer KV projections and changes the architecture’s memory pathway. Teams should therefore validate quality, long-context behavior, batching, and distributed placement on their own workloads rather than assuming that cache reduction will translate directly into end-to-end cost savings.
Why DepthGraft matters
For researchers, the paper offers a useful reframing: memory-sharing connections can shape learning dynamics, not merely reduce inference storage. For practitioners operating long-context LLMs, the reported combination of lower KV memory, faster prefill, nearly unchanged decoding speed, and stronger benchmark performance makes hierarchical KV reconstruction worth testing. The most important design lesson is topology. Where KV is shared, how far apart the layers are, and how gradients are distributed may matter as much as the amount of cache removed.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.