IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation
Text and images rarely appear in isolation. A travel guide combines descriptions with photographs, a recipe pairs instructions with pictures of intermediate steps, and a story uses illustrations to establish characters and scenes. Yet unified multimodal models have generally been evaluated more often on separate text or image tasks than on producing these interleaved experiences as a coherent whole.
In IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation, Shaohan Yu and colleagues argue that progress is constrained by two missing pieces: generation-focused training data and evaluations that explain why one model performs better than another. Their response is IntReal, a dataset and evaluation suite designed around real-world multimodal documents rather than isolated prompts.
A dataset organized around four real-world scenarios
The central training resource, IntReal-100K, contains 100,000 interleaved samples, 538,452 images, and 146.6 million text tokens. The authors collect material from seven web and video sources and organize it into four categories:
- Planning: content such as travel guidance, where text supports decisions and images show places or routes.
- Knowledge: explanatory material that connects concepts with visual references.
- Tutorial: procedurally ordered instructions, such as cooking steps accompanied by images.
- Story: narrative content in which text advances events while images establish characters and settings.
This taxonomy is more than a filing system. It describes different relationships between language and imagery. A tutorial needs visual progression, for example, while a knowledge article may prioritize informative text and illustrative relevance. The authors use that distinction to create a dataset intended for generation, not merely multimodal understanding.
Data curation is deliberately aggressive. The pipeline filters image metadata and visual quality, removes selected watermarked or unsuitable content, deduplicates images within sequences, and uses language and vision-language models to refine captions, surrounding text, prompts, and annotations. Only about 10% of the collected material is retained. The result is a cleaner resource, but also one whose distribution is shaped heavily by automated filters and the chosen source domains.
Seven metrics instead of one opaque score
The IntReal Evaluation Suite uses 600 manually reviewed samples, balanced across the four categories. It evaluates models in two settings. Holistic evaluation asks a system to generate a complete interleaved response. Fine-grained evaluation fixes the preceding context and asks the model to generate the next image, making it easier to diagnose specific failures.
The suite reports seven dimensions:
- Text Richness and Text Logicality assess informativeness, completeness, coherence, and factual plausibility.
- Image Aesthetic measures perceptual image quality.
- Image Diversity penalizes excessive repetition between adjacent generated images.
- Quantity Compliance checks whether the requested or expected image count is met.
- Context Fidelity evaluates whether an image satisfies visual requirements implied by the preceding context.
- Subject Consistency compares generated subjects with reference subjects in settings such as tutorials and stories.
The authors combine language-model judges, vision-language verification, image-quality assessment, and visual feature similarity. They explicitly caution that the equally weighted overall score is mainly a convenient summary; per-dimension results are more informative for understanding a model’s behavior.
What the experiments show
Yu and colleagues report several validation results for the benchmark. The seven metrics have low redundancy: the maximum absolute pairwise Spearman correlation is below 0.42. Automatic judgments agree with human judgments in 83.3% of comparisons, and rankings produced by two different expert evaluators show Spearman correlations above 0.9 for the tested language and context metrics. These findings support the claim that the suite measures several related but distinguishable capabilities, although they do not make model-based evaluation equivalent to human review.
The model comparisons also reveal why a single leaderboard number can mislead. The evaluated unified multimodal models show different strengths. Emu3.5 is strongest among the reported systems on Text Logicality, while SenseNova-U1-8B-MoT performs particularly well on Quantity Compliance and Context Fidelity. The closed-source NanoBanana2 leads on several text-oriented dimensions and image aesthetics, but does not dominate every interleaved-specific measure. The authors interpret this as evidence that strong standalone language or image generation is useful but insufficient: models also need to learn how modalities cooperate across a sequence.
Post-training results are especially relevant. Supervised fine-tuning on IntReal-100K improves the SenseNova-U1-8B-MoT model’s text richness, text logicality, and subject consistency, but reduces image aesthetic quality and context fidelity. Reinforcement learning using image-quality, visual-similarity, and rubric-based rewards recovers much of that loss and improves image diversity. The authors report that the SFT-plus-RL version achieves the highest mean score among the open-source systems they evaluate.
This supports a narrower, practical conclusion: optimizing one modality under multimodal context can transfer to interleaved generation. It does not establish that reinforcement learning is always superior to supervised fine-tuning, nor that the same reward design will generalize to other model architectures or domains.
Limitations and practical implications
The dataset’s strongest feature—strict curation—is also a limitation. Aggressive filtering produces a relatively small retained fraction and may remove legitimate variation, unusual layouts, or examples that are difficult for automated quality models. The four categories cover important use cases, but they do not represent every form of multimodal communication. The evaluation set is compact because each model must generate thousands of images, which keeps testing feasible but limits coverage.
There are also methodological caveats. Several metrics depend on expert models, and the authors note that absolute Text Logicality scores can differ between evaluators even when model rankings remain similar. Reference-based measures such as Subject Consistency may penalize a valid alternative image that differs from the source while still satisfying the prompt. The experiments also do not explore mixed post-training with unimodal data, synthetic augmentation, or broader combinations of training sources.
For practitioners, IntReal suggests evaluating multimodal applications as workflows rather than as collections of isolated outputs. A production system for education, documentation, travel, or commerce should track text quality, image quality, sequence diversity, instruction compliance, and cross-modal grounding separately. Teams fine-tuning unified models should expect trade-offs between language and image behavior, and should test modality-specific rewards or optimization rather than assuming that more joint supervision will solve every failure.
More broadly, IntReal frames interleaved generation as its own capability. The authors’ dataset and benchmark do not settle how it should ultimately be trained or judged, but they provide a concrete way to expose failures hidden by holistic scores—and a useful starting point for building multimodal systems that communicate through coordinated sequences of words and images.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.