JoVox: Joint Hierarchical Flow Matching for Lightweight Voxel Generation

JoVox: Joint Hierarchical Flow Matching for Lightweight Voxel Generation

Generating detailed 3D shapes with voxels involves a difficult trade-off. Dense voxel grids are straightforward to represent, but their cost grows rapidly with resolution because the model processes empty space as well as occupied cells. Hierarchical methods reduce that burden by refining only occupied regions, yet they introduce a different problem: an early, coarse decision determines where all finer geometry can appear. If that decision is wrong, later stages may be unable to recover.

In JoVox: Joint Hierarchical Flow Matching for Lightweight Voxel Generation, Xiangjun Tang, Jianqi Chen, and Peter Wonka of KAUST propose a way to make those early decisions revisable. Their method learns the joint distribution of voxel occupancies across all levels of a hierarchy rather than generating one level completely before moving to the next. The authors report better fidelity than the baselines in both image-conditioned and unconditional generation, while using substantially less computation than the voxel stage of TRELLIS.2.

The problem with level-by-level voxel generation

A voxel hierarchy resembles an octree: a coarse cell can be subdivided into eight children, and those children can be refined again. Conventional hierarchical generators predict the coarse structure first, then condition each finer level on what came before. This is efficient because empty regions are skipped, but it limits the information available when the most consequential decisions are made.

Fine geometry can reveal that a coarse prediction was implausible. In a strictly forward process, however, the model does not revisit that prediction. JoVox addresses this by allowing masked cells at different resolutions to be predicted together. A coarse occupancy can therefore be inferred using finer neighboring cells that have already been revealed.

How JoVox works

The framework builds on masked discrete flow matching. Generation begins with masked occupancy information and progressively unmasks cells. Unlike a fixed sequence of image or language tokens, the voxel hierarchy grows as newly occupied cells reveal their children. The model consequently handles a variable number of cells and can process information from multiple hierarchy levels in one transformer sequence.

The authors introduce two central design choices. First, joint hierarchical modeling replaces separate distributions for each level with one distribution over the hierarchy. During training, different levels are masked independently, exposing the network to many combinations of coarse and fine information. During inference, some fine cells may be known while nearby coarse cells remain uncertain.

Second, two-level prediction skips a level in the network’s processing pattern. Instead of predicting the children one level down, the model predicts eight occupancy codes that describe cells two levels finer. Each code contains eight binary occupancy values, avoiding an impractically large distribution over all 64 cells at once. Because the network processes roughly half as many hierarchy levels, the authors find that this design both accelerates training and improves fidelity in their ablation study.

JoVox also uses an unmasking strategy that revisits old decisions. Occupancy codes unmasked early were predicted with less fine-scale context, so the sampler remasks the oldest codes and predicts them again after additional detail has appeared. This is intended to reduce inconsistencies between codes unmasked in the same step and to give coarse predictions access to later evidence.

What the experiments show

For image-conditioned generation, the authors train on 174,528 assets and test on 1,000 unseen objects from Toys4k. At a finest resolution of 643, JoVox achieves the best value on every reported voxel metric in the paper’s comparison. Against TRELLIS.2, its voxel Chamfer distance is 11% lower, with an IoU of 0.4158 versus 0.4017. When the generated voxels are passed through the shared mesh stage, JoVox also produces a 22% lower shell Chamfer distance than TRELLIS.2.

For unconditional generation, the evaluation uses chair, airplane, table, and rifle categories from ShapeNet. Averaged across the four categories, the authors report the best result on all six listed metrics. Compared with the strongest baseline values, JoVox reduces voxel feature FID by 80% and rendered-voxel FID by 64%. These results support the authors’ claim that jointly revisiting hierarchy levels can improve both distribution quality and geometric fidelity, although the metrics do not eliminate the need for visual and application-specific evaluation.

The efficiency results are particularly notable. On one A100 GPU, with the same backbone as the compared TRELLIS.2 voxel stage, JoVox uses 29.0 GB rather than 47.3 GB at batch size 32 and reports eight-times higher training throughput. Its measured forward-pass cost is 0.82 TFLOPs versus 12.2 TFLOPs, or about one fifteenth as many FLOPs. The authors trained JoVox on four H100 GPUs, compared with 32 H100 GPUs reported for TRELLIS.2, and say it matched or exceeded the comparison quality after substantially shorter training.

Limitations and practical implications

The paper’s results are promising, but they should be read within the stated scope. The main model is evaluated primarily at 643; the authors also show 2563 generation, but those results are presented separately rather than as the main benchmark. The training set contains about 175,000 assets, less than the roughly 800,000 assets used to train TRELLIS.2. More data might improve generalization, but that remains an expectation rather than a demonstrated result.

Independent sampling of multiple occupancy codes can still create inconsistencies. The proposed remasking strategy reduces this issue in the reported ablation, yet it does not model every dependency explicitly. The experiments also focus on single-object generation and selected benchmark categories. Performance on large scenes, multiple interacting objects, or long sequences of shapes remains future work.

For practitioners, JoVox suggests that sparse voxel generation need not choose between hierarchical efficiency and high fidelity. Its approach could make image-to-3D pipelines more accessible when GPU memory, throughput, or training scale is constrained. The broader lesson is methodological: in structured generation, committing early to a coarse representation may be efficient, but allowing later evidence to revise that structure can be worth the added sampler complexity.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close