Open Vision Banana: A Fully Open Recipe for Generative Visual Perception
Image generators are usually judged by what they create. The authors of Open Vision Banana (OVB) ask a different question: can an open image-to-image generator also become a general-purpose vision system that segments objects, estimates depth, predicts surface normals, and still edit or generate images?
The paper presents an open recipe for doing that. Rather than adding separate task-specific prediction heads, OVB represents every task as an RGB image. A prompt tells the model what to produce, the generator creates a visual output, and a deterministic decoder converts that output into a segmentation mask, depth map, or surface-normal field. This is conceptually simple, but the paper shows that the details of the visual encoding and prompt design are decisive.
Turning perception into image generation
OVB fine-tunes open-weight FLUX.2-Klein models at 4B and 9B parameter scales. The transformer is adapted while the VAE and text encoder remain frozen. The training mixture contains public perception datasets plus public replay data for text-to-image generation, image editing, and identity reconstruction. Depending on the regime, the authors use 2.1 million or 3.8 million training pairs.
For semantic segmentation, the target image paints each requested class in a prompt-specified color and leaves other pixels black. The color assignments are randomized rather than permanently associating, for example, cars with one canonical color. Instance segmentation uses a palette of 32 colors and a fixed spatial ordering of instances. Referring segmentation paints the region described by a natural-language expression. Depth is encoded as relative inverse depth in grayscale, while surface normals are mapped into RGB using a camera-space convention.
That common RGB interface allows the model to use the same flow-matching objective across heterogeneous tasks. It also preserves the generator’s original image-to-image interface instead of requiring a collection of specialized heads.
Why colors and prompts matter
One of the paper’s clearest findings is that output conventions are part of the learning problem, not a cosmetic implementation detail. In an ablation, models trained with fixed class-color mappings performed well when evaluated with the same mapping but collapsed when colors changed. Randomized colors taught the model to follow the legend supplied in the prompt instead of memorizing a class-to-color shortcut.
The authors found that a palette of 64 well-separated colors offered the best overall trade-off in their tested settings. Prompt composition also mattered. Mixing prompts that request subsets of classes, all present classes, and absent negative classes helped the model learn both what to paint and what not to hallucinate. Geometry prompts that explicitly described pixel alignment and the required representation were also helpful.
These results suggest a practical lesson for generative perception systems: the model is not merely predicting a mask or a depth field. It is learning a contract involving language, colors, spatial alignment, and a decoder. Breaking any part of that contract can reduce measured accuracy even when the underlying region is located correctly.
Results across perception tasks
On Cityscapes semantic segmentation, the zero-shot OVB 9B model reached 73.9 mIoU with legend ensembling, compared with 69.9 reported for the proprietary Vision Banana system. On COCO-Object, the zero-shot 9B model reached 53.0 mIoU with the same type of test-time consensus, while the non-zero-shot version reached 59.3. The paper reports that these results surpassed the listed specialist baselines under the stated protocols.
Legend ensembling is the paper’s most distinctive inference technique. The model processes the same image several times using different class-color legends, keeps the generation seed fixed, decodes each RGB result into a class map, and combines the maps by majority vote. The authors report gains of up to 8.7 mIoU points on Cityscapes and 15.0 points on COCO-Object without retraining. Their analysis indicates that much of the improvement comes from correcting unstable color-to-class bindings rather than from discovering entirely new regions.
Performance was more mixed elsewhere. The zero-shot 9B model reached 57.4 cIoU on RefCOCOg referring segmentation and 67.1 gIoU on ReasonSeg in the main comparison. Non-zero-shot training improved these results, reaching 69.2 cIoU and 68.6 gIoU for the 9B model. Query rewriting helped reasoning segmentation substantially, but the paper finds that rewriting is most useful after fine-tuning has taught the generator to perform segmentation.
Instance segmentation remained a weakness relative to the strongest proprietary or non-zero-shot systems. OVB 9B achieved 37.4 cgF1 in the zero-shot setting with legend ensembling and 39.6 in the non-zero-shot setting, below the reported Vision Banana and SAM 3 results.
Geometry and retained generation abilities
OVB also produced competitive relative depth and surface-normal predictions. The zero-shot 9B model reached 97.4 percent δ1 accuracy on NYUv2 depth and a 19.85-degree mean angular error on NYUv2 normals. Non-zero-shot training improved NYUv2 results, but it did not improve every dataset: for example, KITTI depth and ScanNet normals showed important trade-offs. The authors emphasize that OVB depth is evaluated after per-image disparity-affine alignment, so it should not be interpreted as absolute metric distance.
The models largely retained image editing. On the authors’ ImgEdit-Bench measurements, the 9B non-zero-shot model matched its base model’s Basic average, although its UGE score declined. Text-to-image quality fell after perception fine-tuning for all four model variants, with the decline smaller at 9B than at 4B. This is evidence of a real trade-off rather than proof that perception tuning preserves generation perfectly.
Limitations and practical implications
The paper’s main limitation is the gap between open and proprietary generators on several perception tasks, especially referring and instance segmentation. Results also depend on evaluation choices, including prompt rewriting, presence gates, deterministic readouts, and multi-pass inference. Legend ensembling improves accuracy but increases computation because every additional pass requires another generation.
For researchers, OVB provides a reproducible baseline for studying unified generation and perception: public data, training code, processing pipelines, checkpoints, and ablations are part of the contribution. For practitioners, the method is most attractive when one model must support both image editing and dense visual analysis, and when a modest inference-time compute budget is available for consensus. The broader message is that generative models can serve as perception systems, but reliability depends on carefully designed output representations and explicit decoding rules. Open Vision Banana makes those normally hidden choices visible—and testable.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.