Cross-modal Steering Vectors Exist Across Both Image and Text Generation
Modern AI models that generate text and images often treat these modalities separately, raising a fundamental question: are the internal concepts and control mechanisms unified across modalities, or are they distinct? A recent study by Donghao Zeng, Nishant Subramani, Tamar Rott Shaham, and Benno Krojer investigates this by exploring cross-modal steering vectors—directions in a model’s internal representation space that can causally steer generation in one modality and transfer zero-shot to steer generation in another.
The Problem: Modality Unification in AI Models
Unified multimodal models (UMMs) generate both text and images using shared parameters, but it remains unclear whether their internal mechanisms truly unify concepts across modalities. Prior work has shown that text and image representations often remain nearly orthogonal inside these models, suggesting separate processing. However, cognitive science proposes a modality-agnostic semantic hub where concepts are shared regardless of input form.
The authors operationalize this debate by asking: can a steering vector found by contrasting minimal pairs in one modality (e.g., text) also steer generation in the other modality (e.g., images) without any adaptation? This goes beyond representational similarity by focusing on causal control directions.
Methodology: Extracting and Applying Steering Vectors
The study focuses on three state-of-the-art UMMs that autoregressively generate text and image tokens from a shared weight space: UniAR, Emu3.5, and Liquid. These models differ in how images are tokenized but share the key property of unified parameterization.
Steering vectors are extracted by generating minimal pairs of inputs that differ only in a target concept (e.g., “a person is happy” vs. “a person is sad” for emotion). The authors generate 100 such pairs per concept in either text or image form. For images, they start from a baseline image and apply targeted edits to produce contrasting poles of the concept.
They then compute the average difference in the model’s hidden states between the positive and negative poles at each layer, normalizing these differences to obtain steering vectors. These vectors are added to the intermediate hidden states during generation, either in the same modality (intra-modality steering) or transferred to the other modality (cross-modal steering).
Key Findings: Cross-modal Steering Works in Both Directions
- Successful transfer: Steering vectors found in text can steer image generation and vice versa, often recovering a large fraction of the intra-modality steering effect. For UniAR, text-to-image transfer recovers on average 84% of the effect, and image-to-text transfer recovers 74%. In Emu3.5 and Liquid, text-to-image transfer sometimes even outperforms image-to-image steering.
- Concept dependence: Concepts that naturally occur in both modalities, such as emotion and age, transfer well. Visual concepts like size and spatial position transfer less effectively, especially image-to-text. The spatial concept (left-right) generally fails to transfer.
- Layer dependence: Transfer is strongest at middle layers for text-to-image steering and varies for image-to-text steering depending on the model.
- Qualitative insights: Steering a text vector for formality causes generated images to depict more formal clothing and design elements. Conversely, steering with an image vector for low resolution causes text outputs to be shorter and more uncertain.
- Geometric signature: Despite a clear modality gap in representations, steering vectors for the same concept in text and image modalities tend to point in similar directions. The closer their cosine similarity, the better the cross-modal transfer.
Evaluation and Evidence
The authors evaluate steering success using a large language model judge that compares steered outputs to unsteered baselines, assessing whether the intended concept is more strongly expressed. They report concept-rate curves across steering strengths and summarize transfer effectiveness as the ratio of cross-modal to intra-modality steering success.
Output quality is also measured: text fluency remains high up to moderate steering strengths before dropping, while image coherence degrades more gently. Cross-modal steering generally maintains output quality comparable to intra-modality steering.
Limitations and Challenges
- Model selection: The study relies on relatively new unified multimodal models with autoregressive generation. Such models are still emerging, and their image generation quality varies, limiting some analyses.
- Concept scope: While many concepts transfer well, some, especially spatial positioning, transfer poorly. Concepts unique to one modality are challenging to define and steer in the other.
- Transfer asymmetry: Text-to-image steering transfers more reliably than image-to-text, possibly due to differences in image tokenization and model architecture.
- Generalization: The findings are demonstrated on three specific UMMs; broader applicability to other architectures or modalities remains to be explored.
Practical Implications and Future Directions
The authors’ findings illuminate how vision and language interact inside unified models, revealing shared causal control directions despite representational modality gaps. This suggests that unified models encode concepts in a modality-agnostic manner to some extent.
Practically, cross-modal steering vectors enable flexible control: one can construct a steering vector in whichever modality is easier or more natural, then apply it to steer generation in the other. This could simplify controlling multimodal generation and improve interpretability.
Moreover, the study opens avenues for modality-agnostic interpretability tools and theories of multimodal interaction. For example, safety interventions or knowledge updates applied in one modality might transfer automatically to others.
The authors encourage extending this work to other unified models and concepts, and investigating the underlying geometry and mechanisms of cross-modal control.
“Our results show that modality-specific representations can nevertheless contain shared causal directions of control, revealing a form of internal structure that is neither fully modality-specific nor captured simply by representational alignment.” — Zeng et al.
In summary, this research provides compelling evidence that cross-modal steering vectors exist and transfer effectively across text and image generation in unified multimodal models, offering new insights and tools for controlling and understanding AI systems that integrate vision and language.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.