PixelModel v6: MMDiT, REPA, and the Capacity Bet
PixelModel v5's README ended with a diagnosis, not a celebration: the model had learned appearance thoroughly and topology poorly, and topology is what capacity buys. v6 is that bet paid out. Six real architecture changes, not one, on top of nearly four times the parameters, on the same data v5 already had working.
Joint attention and real text conditioning
v4 and v5 read text through cross-attention, the way PixArt-alpha does it: image tokens query a fixed text sequence and that is the entire path between the two modalities. v6 concatenates image and text tokens into one sequence and runs joint self-attention over both, the way SD3's MMDiT does it, so text tokens can attend to image tokens and not just the reverse.
The joint stream is also built from google/flan-t5-base hidden states rather than CLIP
token embeddings. CLIP's pooled vector survives, but only for the global adaLN conditioning signal.
T5 conditioning is the specific change PixArt-alpha, DeepFloyd IF, and SD3 all made for prompt
adherence, and it is aimed straight at a failure v5's own README named: prompts asking for one object
in an empty scene often returned the scene without the object.
Position encoding and stability, in service of a bigger model
Image tokens get 2D axial rotary position encoding, text tokens get standard 1D rotary encoding, in place of the additive sincos embeddings v4 and v5 used. RoPE carries no learned parameters, which also means a checkpoint trained at 256px can extend to a larger token grid later without retraining or interpolating a position embedding table.
QK-norm and SwiGLU round out the list: RMSNorm on queries and keys before attention, and a SwiGLU MLP in place of GELU. Both are standard modern stabilizers, and they earn their place here because a bigger model trains less forgivingly than a 40M one.
REPA: the one change without precedent at this scale
An auxiliary loss projects mid-stack image token hidden states to align with frozen DINOv2-small features of the same image, cosine similarity, weight scheduled to peak early and decay to exactly zero by step 105,000. Every other change on this list has direct precedent doing exactly this job at a similar scale. REPA does not: the original paper is class-conditional ImageNet at 675M parameters, and whether the alignment signal transfers to text-conditional generation at less than a quarter of that size was a real open question, not a formality. The decay schedule follows the follow-up finding that the alignment loss helps early and can actively hurt if left on too long.
And more of it
dim 512, depth 16, heads 8: 155.1M trainable, versus v5's 40M. Still well short of SD1.5's 860M UNet or SDXL's 2.6B, but the largest this line has been, and the whole point given v5's diagnosis.
The frozen VAE also moved from sd-vae-ft-mse to
madebyollin/sdxl-vae-fp16-fix, a drop-in upgrade at the same latent shape.
Training
150,000 steps at batch 192 on the same 2,986,571 image CC12M-recaptioned set v5 used, about 9.7 epochs. AdamW at 2e-4, cosine decay after a 1,500 step warmup, EMA at 0.9999, 10% caption dropout, gradient checkpointing throughout to fit the bigger model in memory. REPA ran at peak weight for the first 40% of training, decayed linearly through the next 30%, and sat at exactly zero for the final 30%, partly the early-stopping finding above and partly because computing DINOv2 targets costs real throughput: the run held 0.86 steps/s while REPA was active and 1.39 steps/s once it turned off.
Held-out loss on a fixed 4,096-pair split, evaluated every 2,000 steps, fell without interruption from 0.938 to 0.798 across the entire run. No overfitting turn-up at any point, the same clean shape v5 reported.
Results
Measured on 5,000 MS-COCO val2014 images at 256 center crop, using torchmetrics FID at
2048 features and torchmetrics CLIPScore with openai/clip-vit-base-patch32.
Identical protocol to v4 and v5.
v4 v5 (cfg 5.0) v6 (cfg 3.0)
FID ↓ 39.54 34.06 23.62
CLIP score ↑ 28.04 29.27 30.43
trainable params 40,013,980 40,013,980 155,090,448
real photographs, CLIP n/a 30.05 30.05 (reference ceiling, all runs)v6 beats v5's best FID by over ten points, and it does it at every guidance scale tested, not just at one cherry-picked value:
cfg 2.0 3.0 4.0 5.0 6.0 7.0 8.0
FID ↓ 25.65 23.62 23.77 24.42 25.10 26.26 26.87
CLIP score ↑ 29.65 30.43 30.81 31.06 31.12 31.16 31.17FID bottoms at cfg 3.0 and rises steadily past it, the usual guidance tradeoff: more weight on the text condition pulls generations away from the real-image manifold even as it pulls them closer to the prompt. cfg 3.0 is the reported default.
Worth flagging rather than celebrating: CLIP score crosses the real-photograph baseline of 30.05 starting around cfg 4.0 and keeps climbing to 31.17 at cfg 8.0. That is not v6 out-photographing real photographs. It is high guidance making generations more literal about the prompt than an actual photo ever is, which CLIP score rewards and FID does not. Use FID, not CLIP, to pick a guidance scale.
What it does well and what it does not
Judged across several hundred samples during training, not just the checkpoints above. It is good at material, light, and increasingly at structure that v5 could not hold: food, faces of common animals, latte foam texture, boat masts and sails, neon-lit streets. Where v5's README singled out faces and animal heads as a weak point, v6 renders recognizable dog and fox faces reliably by the back half of training, something no earlier version in this line managed.
It is still uneven on rare subjects. Across many samples of the same fox prompt, output quality varied far more than for common subjects like coffee or strawberries, which is a direct read on training data frequency in CC12M rather than a model defect: the architecture change did not fix data scarcity, it just makes better use of what is there.
A watermark and ghost-text artifact shows up intermittently, most often on the lighthouse and boat prompts: faint illegible text-like marks bled in from CC12M's watermarked stock photography, the same failure family v5 documented for actual text rendering. It did not fully train out by step 150,000.
Specification
trainable parameters 155,090,448
frozen VAE (sdxl-vae-fp16-fix) 83,653,863
frozen text encoder (CLIP ViT-B/32) 63,165,952
frozen text encoder (T5-base) 109,628,544
combined total 411,538,807
architecture MMDiT, dim 512, depth 16, heads 8, patch 2
tokens 256 image + 32 text
objective rectified flow, target x1 - x0, logit-normal timesteps
conditioning joint attention on T5 tokens, plus adaLN-zero from CLIP pooled + timestep
resolution 256x256
sampling 50 steps, cfg 3.0 default (23.62 FID)All parameter counts above are the released backbone. Training also carried a small REPA projection head, 459,648 params, 155,550,096 total while training, that only ever fed the auxiliary alignment loss. It has no role in sampling and is dropped from the published weights.
Try it
git clone https://huggingface.co/bench-labs/PixelModel-v6
python main.py "a bowl of ramen with a soft boiled egg" --out ramen.png --cfg 3.0 --steps 50
As with every model in this line, the weights are also stored as a PNG. model.png is not
a picture of the network, it is the network, with each parameter packed across the red and green
channels of one pixel. Explore the
PixelModel v6 repository or the
text-to-image leaderboard,
where it now sits at the top of the board on FID.
v5 diagnosed the ceiling: appearance was learned, topology was not, and topology is what capacity buys. v6 spent the capacity and the diagnosis held: FID 23.62 against v5's 34.06, at cfg 3.0, on the same data, the same eval protocol, no cherry-picking.