VoxelModel v1: The Decoder Had to Go

Every model in this line so far has been a coordinate decoder. PixelModel v1 answered "given this prompt and the coordinate (x, y), what colour is here?", and v2 through v4 kept that shape while making the question better. So the obvious move for 3D was to add a z, ask the same question, and ship a SIREN that paints occupancy into a cube.

We tried that. It does not work well, and the reason is boring: a coordinate decoder has no cheap way to know what the rest of the shape is doing. Colour in a 2D image is mostly a local question. Whether a voxel is solid depends on whether the chair it belongs to has four legs, and the decoder answering point by point cannot see the other three.

So VoxelModel v1 breaks the pattern. It is a diffusion transformer over the whole grid at once, 40,050,892 parameters, trained for 9.7 hours on a single A100. It still ships as a PNG, because that part was never about the architecture.

The architecture

A 32³ grid gets cut into 4³ patches, which gives 512 tokens. Those go through a standard DiT stack with cross-attention on frozen CLIP ViT-B/32 text features, plus adaLN-zero modulation from the pooled text embedding and the timestep. The training objective is rectified flow.

prompt string
  -> frozen CLIP ViT-B/32 (40 tokens x 512)   [37.8M, not trained]

32x32x32 occupancy grid
  -> patchify at 4      = 512 tokens x 384
  -> + 3D sincos positional encoding
  -> 12 x [ self-attn | cross-attn(text) | MLP ], adaLN-zero
  -> unpatchify         = 32x32x32 velocity

target = x1 - x0,  t = sigmoid(randn)

There is no VAE anywhere in that stack. The model reads and writes raw occupancy, so the whole thing is 40.05M trained plus 37.8M frozen text encoder, 77.9M total. PixelModel v4 was 161M for a comparable trained size because it carries an 83.7M sd-vae-ft-mse. This is the first model in the line that is small without an asterisk.

Half the dataset is unusable, and the caption tells you which half

Meshes come from dylanebert/objaverse-lowpoly-obj, captions from Cap3D, joined on Objaverse UID for 58,374 captioned meshes. The problem is that a lot of those "meshes" are several objects sitting in one file. Normalization scales the whole scene bounding box into the grid, so every extra object shrinks the others, and 32³ resolves a cluster of blobs instead of a thing.

The useful part is that the caption gives this away before you load any geometry. A mesh captioned "a red and white bench with a pink variation" is two benches. So caption_filter.py rejects scene and variant-set wording, a top level and joining two noun phrases, and captions over 12 words. Colour pairs are kept on purpose, so "a purple and green mushroom" survives and "a white cube and a white robot" does not.

Over all 58,374 captions, 29,439 survive, which is 50.4%. The rejections break down as 29.1% for length, 16.6% for a top level and, and 3.9% for explicit scene wording.

One thing worth knowing if you build something similar: occupancy does not detect this defect. We checked, expecting multi-object meshes to be denser. Over a 900 mesh sample the survivors and the rejects have medians of 4.3% and 5.0%. No threshold on the geometry would have found it. The signal is in the caption and in a render, and nowhere in the summary statistics.

Training

90,000 steps at batch 256 over 28,415 meshes, with 1,024 held out. AdamW at 2e-4, betas (0.9, 0.99), gradient clip 1.0, cosine decay to 1e-6 after a 500 step warmup, EMA 0.9999, and 10% CFG dropout. It held 2.57 steps per second at 41.4 GiB the whole way.

The held-out curve is the reason we bothered holding data out. It bottoms at 0.018737 at step 64,000 and then drifts back up to 0.019314 by step 90,000. That is 3.08% of overfitting, and it means roughly the last 2.8 hours of the run were spent getting slightly worse. 90,000 steps over 28,415 meshes is 811 epochs with no augmentation, so this is not surprising, but we would not have known without the split.

We also shipped the wrong checkpoint. Saves went to a fixed path and overwrote each other, so the step 64,000 weights no longer exist and the release is the step 90,000 ones. Worth 3.08% to anyone who retrains.

What it actually beats

Evaluating text-to-3D properly is unsolved and we are not going to pretend otherwise. FID does not transfer. What we can do is generate from the 1,024 held-out captions, compare against the real mesh each caption came from, and put that next to baselines that are allowed to win.

Two things fall out of that. The first is good: stripping the prompt drops IoU from 0.1898 to 0.0489, which is where a randomly chosen training mesh sits (0.0516). The text conditioning is doing essentially all of the work, and the model is not just producing an average blob that happens to overlap everything.

The second is not flattering. Nearest-neighbour retrieval beats us on IoU. Taking the training mesh whose caption is closest in CLIP space scores 0.2006 against our 0.1898. We win on Chamfer, 4.165 against 4.487, which says the generated shapes are geometrically closer on average even when the voxel overlap is not, but that is a tie at best. A 40M parameter model trained for 9.7 hours is roughly as good at this benchmark as looking up the closest caption you already have.

In fairness to the model, IoU against one specific mesh punishes valid diversity, and retrieval gets to return a real object every time. But we are not going to round that up into a win.

The failure mode is density, not detail

Our first guess was that thin objects break because a sword cannot exist at 32³. That guess was wrong, and the data says so twice.

The real swords in the dataset voxelize fine. The one below sits at 2.9% occupancy and is clearly a sword. Swords are not rare either: 1,159 training captions mention one, which is more than chair (541) or tree (537).

What actually goes wrong is that the model compresses the range of densities toward the middle. Split the held-out set by how much of the grid the real mesh fills, and the thinnest quarter has a true occupancy of 1.24% while the model generates 3.58% for those same prompts, 2.9 times too solid. The chunkiest quarter is nearly right, 11.08% true against 12.83% generated. Correlation between true and generated occupancy is 0.565, so it has partial control and not much more.

The thinnest quarter is clearly the worst bucket. We will not claim more than that, because the middle two are not ordered the way a clean story would want them, and with 256 meshes per bucket we are not going to read much into the gap between 0.2378 and 0.2295.

This is a more fixable problem than "the resolution is too low". A loss that does not treat an empty voxel and a solid voxel as symmetric errors would help, and so would sampling that is not free to drift toward the mean.

What is missing

• Colour. These meshes carry no vertex colours and no .mtl files, so v1 generates grey geometry only. Colour needs a source with materials and its own voxelization path.
• A real metric. IoU and Chamfer against a single ground truth mesh are the best we have and they are not good enough. Rendered FID with a CLIP score over multiple views is the obvious next thing to build.
• Augmentation. Yaw rotations and a mirror are 8x more data for free on a cubic grid, and 811 epochs is exactly the situation that wants it.

Try it

git clone https://huggingface.co/bench-labs/VoxelModel-v1
cd VoxelModel-v1
python sample.py "a wooden chair" "a purple mushroom" --cfg 5.0 --steps 50

That writes a JSON of base64 np.packbits grids. As with every release in this line the whole network is also in model.png, a 6329x6329 image holding all 40,050,892 weights, which round-trips against the safetensors to 9.7e-04. Everything is at the VoxelModel v1 repository.