Hy-Nomas: Advancing Free-form Interactive World Generation
Interactive world generation models aim to create evolving virtual environments that respond dynamically to user commands. Unlike traditional video generation, which produces fixed clips from static prompts, interactive world models must handle successive inputs, maintain historical context, and ensure that actions have consistent consequences over time.
The Challenge of Interactive Video World Models
Building a model capable of free-form interaction involves several complex requirements. The system must:
- Interpret and execute arbitrary text instructions issued at multiple points during generation.
- Maintain persistent scene information and object identities across successive video segments.
- Support continuous camera control, allowing users to navigate the scene fluidly.
- Prevent temporal information leakage that could break the causal flow of events.
- Handle compounding errors from autoregressive generation to avoid visual drift or implausible motions.
These challenges necessitate innovations in model architecture, training strategies, data preparation, and inference optimization.
Introducing Hy-Nomas
Hy-Nomas is an 8-billion parameter video world model developed by Tencent’s Hy Visual Generation Team. It builds on a pretrained bidirectional diffusion transformer (DiT) backbone and adapts it for multi-turn, free-form interactive world generation with integrated camera navigation.
Key features of Hy-Nomas include:
- Segment-wise autoregressive generation: The model generates variable-length video segments conditioned on current inputs and cached history, ensuring temporal causality.
- Action-aware causal attention: A novel attention masking scheme distinguishes between persistent descriptive context and temporally localized action instructions, reducing errors such as action repetition or misalignment.
- Vision-language Prompt Enhancer (PE): A dedicated vision-language model translates user requests and multimodal interaction histories into structured prompts and segment durations for the next generation step.
- Continuous camera control: Camera trajectories are encoded as relative frame-to-frame motions and injected via cross-attention, enabling fine-grained navigation alongside text instructions.
- Progressive multi-stage training curriculum: The model is gradually adapted from bidirectional video generation to causal autoregressive generation, action alignment, camera conditioning, segment forcing on model-generated histories, and finally diffusion distillation for efficient inference.
Data and Captioning Strategy
Hy-Nomas’s success relies heavily on a large, heterogeneous video corpus spanning real and virtual environments. The data pipeline includes:
- Standardized preprocessing and filtering for video quality and format consistency.
- Hierarchical clustering and semantic deduplication to remove redundant clips.
- Taxonomy labeling using a vision-language model to categorize videos by content and visual attributes.
- Camera trajectory estimation and filtering using a specialized pipeline to provide geometric supervision.
- Structured multi-turn video captioning that maintains subject identities, timestamps actions, and interleaves scene and camera descriptions chronologically.
This rich annotation enables the model to learn precise temporal alignment between actions, scene changes, and camera motions, which is critical for interactive control.
Model Architecture and Training
Hy-Nomas extends the HunyuanVideo 1.5 architecture with:
- Segment-wise causal attention: Attention is bidirectional within segments but causal across segments, preventing future information leakage.
- Action-aware masking: Within each segment, action-specific text tokens are restricted to their temporal windows, improving alignment and preventing repeated execution of past actions.
- Camera cross-attention: Visual tokens attend to temporally aligned camera tokens encoding relative camera poses.
The training curriculum is carefully staged:
- Latent-space adaptation: Transfer from the original VAE to a more compressed latent space for efficiency.
- Segment-causal continual pre-training: Establish autoregressive generation with segment-wise causal attention.
- Action-aware continual pre-training: Introduce temporally localized action supervision and extend context length.
- Camera-conditioned continual pre-training: Add continuous camera control with mixed text and camera conditioning.
- High-quality annealing: Fine-tune on high-quality videos with detailed captions and camera trajectories.
- Segment forcing: Train on model-generated histories to mitigate compounding errors during autoregressive inference.
- Diffusion distillation: Reduce sampling steps for real-time generation while preserving quality.
Inference and Adaptation
For deployment, Hy-Nomas incorporates:
- Prompt Enhancer (PE): A vision-language model that converts user instructions and interaction history into structured prompts for the video generator, supporting multi-turn interactions and both text and camera control modes.
- Inference optimizations: INT8 quantization, sliding-window attention, CPU offloading, and sparse attention enable efficient generation on a single NVIDIA RTX 4090 GPU.
- Pixel-space video decoder: A diffusion-based decoder restores high-frequency details and realistic textures, overcoming artifacts from highly compressed latents.
- LoRA-based downstream adaptation: Users can fine-tune lightweight adapters on the base autoregressive model and merge them into the distilled model for customized visual styles or domain-specific tasks without retraining the entire model.
Evaluation and Results
The authors introduce Hy-Nomas-Bench, a comprehensive evaluation suite with 280 test cases spanning four benchmarks:
- Multi-turn free-form text interaction: Tests instruction execution consistency over successive turns.
- Camera navigation: Assesses response accuracy to keyboard-based camera controls and reversibility.
- Interleaved text and camera control: Evaluates model performance when switching between semantic instructions and camera navigation.
- Spatial memory: Measures consistency of object appearance and scene layout when revisiting previously observed viewpoints.
Human evaluations comparing Hy-Nomas with state-of-the-art models such as Genie 3, LingBot-World v2, and Happy Oyster demonstrate that:
- Hy-Nomas consistently outperforms competitors in prompt following and physical plausibility during multi-turn text interaction.
- It achieves superior or comparable performance in camera control tasks.
- Hy-Nomas uniquely supports interleaved text and camera control within a single unified model, with strong performance.
- Spatial memory remains a challenge; while Hy-Nomas performs comparably to Happy Oyster, Genie 3 maintains stronger long-term spatial consistency, possibly due to grounding in real-world data.
LoRA adaptations show that Hy-Nomas can be effectively fine-tuned for cinematic visual styles or robotics tasks, improving visual quality or out-of-distribution performance with minimal data and training.
Limitations and Future Directions
While Hy-Nomas advances interactive world generation, the authors acknowledge several limitations and open challenges:
- Spatial memory: Maintaining consistent object states and scene layouts over long rollouts remains difficult, especially when revisiting locations after complex camera motions.
- Physical plausibility: Although improved, ensuring that generated actions correspond to physically executable outcomes (e.g., robotic grasps) requires further integration with action-conditioned data and evaluation.
- Inference efficiency: Despite optimizations, real-time responsiveness for gaming or robotics applications demands continued advances in distillation, quantization, and memory management.
- Memory management: Scaling to longer interactions may require hierarchical or retrieval-based memory systems that selectively store and recall relevant scene information.
- Data and caption quality: The model’s performance heavily depends on detailed, temporally aligned captions; improving captioning and prompt engineering remains crucial.
Practical Implications
Hy-Nomas represents a significant step toward versatile, free-form interactive video world models that can be controlled through natural language and camera navigation. Potential applications include:
- Gaming: Generating dynamic, richly detailed game environments responsive to player commands and camera movements.
- Robotics: Simulating realistic robot interactions and visual outcomes for training and planning.
- Virtual production and storytelling: Enabling creators to direct scenes interactively with natural language and camera control.
- Research: Providing an open-weight, reproducible platform for advancing interactive world modeling and evaluation.
By releasing model weights, code, benchmarks, and adaptation workflows, the authors facilitate further research and practical experimentation in this emerging area.
Conclusion
Hy-Nomas demonstrates that combining structured temporal supervision, action-aware causal attention, continuous camera conditioning, and a progressive training curriculum can produce an 8B-parameter video world model capable of free-form, multi-turn interactive generation. Its superior performance across diverse benchmarks highlights the promise of this approach, while identified challenges point to rich avenues for future work in memory, physical grounding, and efficiency.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.