PixelModel v5: The Bottleneck Was Data, Not Architecture

PixelModel v4 gave the series real text understanding: a Transformer text encoder, spatial cross-attention, native 256×256 output. It was a genuine architectural jump, and it still scored FID 39.54. v5 changes none of that. Same DiT, same width, same depth, same VAE, same text encoder, same rectified-flow objective, same 40,013,980 trainable parameters. The only thing that moved is the data, and that alone was worth an FID drop to 34.06.

v4 was starving, not undersized

v4 trained on MS-COCO, about 83,000 images. At batch 256 for 80,000 steps that is roughly 247 passes over the same photographs. A model that sees the same 83,000 images 247 times is not learning what the world looks like, it is memorising a small album.

We looked at architecture changes first, the way you always do. Linear attention, deeper stacks, a higher compression autoencoder, representation alignment through REPA. All real techniques, and none of them address the actual problem, because the bottleneck was never the shape of the network.

v5 trains on 2,986,571 image and caption pairs, about 10 epochs instead of 247.

The captions matter as much as the count

The images come from CC12M. Raw CC12M alt text is close to unusable for this, because it is scraped SEO copy — a typical caption reads like "This is the best recipe I have ever tried for Cuban bread. Cuban Recipes, Bread Recipes, Cooking Recipes, Pan Cubano". None of that tells a diffusion model what is actually in the frame.

v5 uses a recaptioned version where a vision model rewrote every caption in COCO style:

A purple-lit subway train sits at a station platform.

An old red tractor with gold fenders and red wheels stands in a snowy field.

A compact kitchenette in a studio apartment features a stainless steel refrigerator, a microwave, and a sink with a glass faucet.

Those describe what is in the frame, which is what a text-to-image model actually needs. They also match the caption distribution that FID and CLIP score are computed against, so the training signal and the evaluation are speaking the same language.

Training

120,000 steps at batch 256 on a single A100, about 7 hours. AdamW at 2e-4 with betas (0.9, 0.99), no weight decay, gradient clipping at 1.0, cosine decay to a tenth of peak after a 1,000 step warmup, EMA at 0.9999 starting after warmup, and 10% caption dropout for classifier-free guidance. It held 4.6 steps per second at 44.2 GiB the whole way.

The held-out curve climbs for the first 10,000 steps or so, which looks alarming and is not. EMA at 0.9999 has a time constant of roughly 10,000 steps, so early on those weights are still mostly initialisation. The number only becomes meaningful once EMA has caught up, and from there it falls without interruption.

v4 had no validation split at all, so it shipped whichever weights the last step happened to produce. v5 holds out 4,096 pairs, evaluates every 2,000 steps, and writes the best checkpoint to a separate file. Across the whole run the held-out loss improved at every single check and never once turned upward — the clearest evidence that the data change worked, not just a side effect of training longer. Our sibling model VoxelModel v1, trained on 28,415 meshes for 811 epochs, peaked at step 64,000 and got 3.08% worse by the end. Same recipe, same team, different data regime, opposite outcome.

Results

Measured on 5,000 MS-COCO val2014 images at 256 center crop, using torchmetrics FID at 2048 features and torchmetrics CLIPScore with openai/clip-vit-base-patch32. This is v4's protocol, run again rather than assumed.

                         v4            v5 (cfg 5.0)   v5 (cfg 3.0)   v5 (cfg 6.0)
FID              ↓       39.54         34.06          35.03          35.18
CLIP score       ↑       28.04         29.27          28.59          29.28
training images          83,000        2,986,571      2,986,571      2,986,571
epochs over data          247            ~10            ~10            ~10
trainable params         40,013,980    40,013,980     40,013,980     40,013,980
real photographs, CLIP    —             30.05 (reference ceiling, all runs)

FID is 13.9% lower and CLIP score is 1.23 points higher, at an identical parameter count. Nothing in the network changed between v4 and v5, so the entire difference is the training data. Feeding real COCO photographs and their own captions through the same CLIPScore harness gives 30.05, which puts v5 at cfg 5.0 within 0.78 of the ceiling this metric can award.

Guidance scale was swept rather than assumed. FID bottoms out at cfg 5.0 and CLIP keeps climbing slightly to cfg 6.0, the usual tension between the two. Sample count matters more than people expect here too — the same cfg 5.0 checkpoint scores 74.35 on 1,000 images and 34.06 on 5,000, so FID numbers are only comparable when the sample count matches.

Specification

trainable parameters                40,013,980
frozen VAE (sd-vae-ft-mse)          83,653,863
frozen text encoder (CLIP ViT-B/32) 63,165,952
combined total                      186,833,795
architecture                        DiT, dim 384, depth 12, heads 6, patch 2
tokens                               256, over a 32x32x4 latent
objective                            rectified flow, target x1 - x0, logit-normal timesteps
conditioning                         cross attention on 40 CLIP text tokens, plus adaLN-zero
resolution                           256x256
sampling                             50 steps, cfg 5.0 for best FID, cfg 6.0 for best CLIP

One correction worth flagging: v4's card reported 161.5M total parameters, but that count left out the CLIP text encoder's 49,408 × 512 token embedding table — another 25.3M parameters that the encoder cannot run without. The corrected total for both v4 and v5 is 186,833,795.

What it does well and what it does not

After a few hundred samples the boundary is consistent. It is good at material and light — food, landscapes, skies, water, snow, foliage, stone, interiors, depth of field. A bowl of ramen, a wet cobblestone street, a wheat field under storm light, a library of wooden shelves. These come back looking like photographs.

It is bad at structure. Faces and animal heads deform. Limbs merge. Wings on aircraft are stubby or fused to the fuselage. Any text it renders, on a shop sign or an airliner livery, is convincing looking gibberish. Prompts asking for one isolated object in an empty scene often return the scene without the object — "a lighthouse on a cliff at sunset" gives a very nice sunset over a cliff and no lighthouse.

Frequency in the training data matters more than complexity. Aircraft are structurally complicated and come out fine because CC12M is full of them. Hot air balloons are simpler and fail, because they are rare.

That failure profile is what a 40M model looks like once data is no longer the constraint. It has learned appearance thoroughly and topology poorly, and topology is what capacity buys. v6 is where that gets spent.

Try it

git clone https://huggingface.co/bench-labs/PixelModel-v5
python main.py "a bowl of ramen with a soft boiled egg" --out ramen.png --cfg 6.0 --steps 50

As with every model in this line, the weights are also stored as a PNG. model.png is not a picture of the network, it is the network, with each parameter packed across the red and green channels of one pixel. Explore the PixelModel v5 repository or the text-to-image leaderboard.

v4 proved the network needed to understand text. v5 proves that once it does, the next several points of FID are bought with data, not layers.