Read, Allocate, Route: Role-Coupled Tokenization for Pixel-Space Generative Transformers
Pixel-space image generators avoid the compression bottleneck of a separately trained autoencoder, but they inherit a costly representation problem: the Transformer must process the full-resolution image. In a conventional design, each image patch becomes a token, and the patch grid determines both how much detail the model can represent and how many tokens participate in global attention.
The cost rises sharply as patches become smaller. For a 256 × 256 image, moving from 16 × 16 patches to 8 × 8 patches increases the token count from 256 to 1,024, making the quadratic portion of dense attention 16 times larger. The authors of Read, Allocate, Route argue that this coupling is unnecessary. Fine detail may need additional representation capacity, but it does not necessarily need the same amount of image-wide communication as low-frequency structure.
The core idea: separate detail from global communication
The paper introduces SpecSlot, a pixel-space rectified-flow Transformer organized around three operations: Read, Allocate, and Route.
- Read: the model divides the input into spatial anchors that collect local image evidence.
- Allocate: each anchor expands into multiple indexed slots. Every slot owns a disjoint group of coefficients in a discrete cosine transform, or DCT, representation.
- Route: only the slot responsible for the lowest-frequency group from each anchor enters global attention. All slots still interact locally within their anchor.
This creates two independent allocation controls. The number of anchors, represented by W, controls how densely the model reads the image. The number of slots per anchor, K, controls how finely each local output is subdivided spectrally. Increasing spectral detail therefore does not automatically lengthen the global attention sequence.
The lowest-frequency slot acts as a carrier for broad structure, which the authors expect to benefit most from cross-image communication. Higher-frequency slots remain local because they primarily model fine detail. SpecSlot also includes a spatial correction head that uses the same slot states to add a local pixel-space prediction. The final state, target, and training loss remain in pixel space rather than being transferred into a latent representation.
What the experiments show
The authors evaluate SpecSlot on class-conditional ImageNet generation at 256 × 256 and 512 × 512, then test text-to-image generation at 256 × 256. Their comparisons use JiT models and a FLOPs-matched JiT control, with reported results averaged across independently trained seeds for the main evaluations.
At ImageNet-256, the W256/K4 version reaches a mean FID of 6.59, compared with 6.68 for the compute-matched JiT control. The paper describes the final quality as comparable while also observing faster early-to-mid training convergence. The result is not a universal win over every baseline: JiT-B/8 records a lower FID of 4.34, but at substantially higher compute.
The larger difference appears at ImageNet-512. SpecSlot W256/K4 reports an FID of 9.14, versus 11.98 for the resolution-matched compute control, with 135.64 GFLOPs compared with 141.21. It trails JiT-B/16, which reports an FID of 8.45, by 0.69 while using 38.2% fewer GFLOPs. The authors also report higher throughput for SpecSlot in this comparison.
For text-to-image generation, SpecSlot scores 62.21 on DPG-Bench, compared with 61.15 for JiT-B/8, while using 129.86 GFLOPs rather than 218.77. That is a 1.06-point improvement at 40.6% lower forward cost according to the paper. On GenEval, however, JiT-B/8 remains ahead, scoring 29.78 compared with SpecSlot’s 26.99. This split is important: the method’s advantage depends on the evaluation metric and is not a blanket improvement in text-image alignment.
Evidence for the routing design
The ablations support the paper’s central allocation argument, though they do not prove that every part of the design will transfer unchanged to other settings. At a fixed anchor grid, increasing the number of slots generally improves FID in the tested range, but with diminishing returns. At a fixed total slot budget, allocating more capacity to spatial anchors performed better than allocating most of it to spectral subdivision in the reported experiments.
Routing also mattered. The GGLL schedule—two global blocks followed by two local blocks, repeated three times—performed better than an otherwise comparable GLGL schedule. A compute-matched dense-global control had a mean FID of 6.81, versus 6.59 for GGLL, while the full-global model reached 5.84 at much higher cost. The authors report that GGLL was 1.47–1.49 times faster than the dense-global control across the tested sampling budgets.
Further diagnostics found that the lowest-frequency group became distinguishable from noise earlier in the rectified-flow trajectory than higher-frequency groups. Replacing the lowest-frequency carrier with another slot worsened FID in every tested case. These measurements are consistent with the proposed role assignment, but they are mechanistic probes and should not be treated as a complete causal explanation for the end-to-end results.
Limitations and practical implications
SpecSlot’s evidence is promising but bounded. The reported evaluations focus on ImageNet and one text-to-image setup, with selected resolutions, training recipes, and model configurations. The strongest comparisons rely on specific JiT baselines and an implementation-defined compute-matched control, so results may change under different architectures, hardware, samplers, or training budgets. The authors also explicitly avoid claiming that the observed preference for more anchors generalizes beyond the tested allocation range.
There is a practical trade-off as well. SpecSlot still evaluates all slots, even though only carrier slots receive global attention. Local computation, output projections, DCT synthesis, and overlap-add decoding remain part of the workload. Consequently, reducing the global quadratic term does not make the entire model cost independent of detail. The measured gains depend on efficient implementation and on the balance between anchor density and spectral subdivision.
For researchers building pixel-space generative Transformers, the paper offers a useful design principle: token count need not be the only allocation axis. Spatial anchors can determine where the model reads, spectral slots can determine what it predicts, and communication can be assigned according to those roles. For practitioners, the immediate lesson is more cautious: SpecSlot may offer a better quality–compute point when high-resolution detail matters, but its value should be validated against deployment hardware, the target resolution, and the evaluation metric that matters for the application.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.