PixelModel v2 and v3: Scaling Up, Then Doing It Right
PixelModel v1 shrank the model to 23,747 parameters, moved to a coordinate-conditioned decoder, and beat the original on both benchmark metrics. The obvious next move was to make it bigger. That's v2. It turned out to be the least interesting release in the series, and we're not going to pretend otherwise — which is exactly why v3 exists.
v2: scaled up, and that's the whole changelog
PixelModel v2 takes v1's architecture — hashed trigram embedding, tanh MLP decoder, Fourier-featured coordinates — and widens it. 23,747 parameters became roughly 200,000. Nothing else changed. The model card says so directly: "23k parameters -> 200k parameters (thats the only change)." Same training recipe, same data pipeline, same decoder shape, just wider layers.
Scored on the same pipeline as v1 and v3, v2 comes in at FID 390.68 and CLIP 20.48 — better than v1's 420.75 / 20.10, for a model that's just wider, nothing else. So width alone does help. The question v2 leaves open isn't "does it work," it's "is this the efficient way to spend the next parameter" — which is what v3 goes after.
v3: a different architecture, not a bigger one
v3 answers that question by being the release that finally gets scored properly, and by not just scaling up again. It replaces three specific parts of the v1 design instead of widening them:
- Learned word embeddings. v1 hashed the prompt into a fixed vector with zero learnable parameters — the text path could never learn what a word means. v3 has a real embedding table (3,923 words, mean-pooled over tokens) that trains with everything else.
- A SIREN decoder. tanh smears high frequencies. v3 uses sine activations with proper SIREN initialization (w0 = 30), which is finicky enough that
train.pyprints per-layer activation stats early in training just to catch a bad init before it wastes a run. - FiLM conditioning. Instead of concatenating the prompt latent onto the coordinates, the latent generates a per-layer scale and shift that modulate each decoder layer. The FiLM generator is zero-initialized, so at step 0 it's a no-op and the SIREN statistics start clean.
The result is 919,427 parameters — about 39× v1, about 4.6× v2 — packed into a 959×959 PNG:
The regression that makes the comparison fair
Comparing v3 to v1's published FID would have been cheating. v1's 439.46 and v3's number would come from different eval code and different library versions — that gap is partly library drift, not model quality. So every model got re-scored through the same pipeline: same 5,000-image MS-COCO val2014 subset, same torchmetrics versions, same center crop.
v1 v2 v3
params 23,747 ~200,000 919,427
FID published 439.46 not published —
CLIP published 20.02 not published —
FID, this pipeline ↓ 420.75 390.68 383.91
CLIP, this pipeline ↑ 20.10 20.48 20.73
model.png 160x149 scaled 959x959 (1.78 MB)
native resolution 64 64 128Re-scoring v1 under the current pipeline gave 420.75, not the published 439.46 — an ~19-point gap from library and subset drift alone, which is why skipping the regression wasn't an option. v2's own numbers (FID 390.68, CLIP 20.48) came from the same pipeline. On identical footing, the ranking holds cleanly in both directions — FID: v3 < v2 < v1, CLIP: v3 > v2 > v1. v3 wins, but so does v2's raw width, just less efficiently.
The bigger lever wasn't the architecture change alone — it was matching train and inference scale. v3 is trained on whole 128×128 frames instead of zoomed-in crops, because a network trained on crops learns patch statistics it never sees when painting a full image. Closing that gap moved FID from 392 to 384 and CLIP from 19.9 to 20.7 without touching a single layer.
Training
v3 trained for 80 epochs on a single RTX 3090, about 40 minutes, batch size 32, cosine learning rate from 2e-4, mixed precision. Loss went from 0.0720 to 0.0540 and had flattened out by the end:
Going bigger doesn't help
We also trained a ~3.4M parameter variant — wider, deeper, 256×256 — to see if more capacity was free upside. It wasn't. At the base learning rate it wouldn't descend at all; even after dropping the learning rate, its best loss (0.070) never reached the 919K model's 0.054 before training destabilized. A deeper SIREN behind a large FiLM hypernetwork is just harder to optimize at this scale, and the extra parameters bought nothing. The released model is the 919K version. That's the actual result, not the one we'd have preferred.
Architecture
prompt string
lowercase / whitespace tokenize -> token ids (up to 20)
learned embedding table (3923 x 64), mean pool over real tokens [251,072]
Linear(64 -> 256) -> sin
Linear(256 -> 192) = latent z (192)
FiLM generator (zero init, so scale=1 / shift=0 at start):
z -> Linear(192 -> 2 x 256 per sine layer) = (scale, shift) x4 [395,264]
for every pixel (x, y), decoded as one batched tensor (no python loop):
Fourier features of (x, y): [x, y] plus sin/cos over 8 octaves = 34 dims
SineLayer 0: Linear(34 -> 256) -> FiLM -> sin(30 * .) (SIREN first layer)
SineLayer 1: Linear(256 -> 256) -> FiLM -> sin(30 * .)
SineLayer 2: Linear(256 -> 256) -> FiLM -> sin(30 * .)
SineLayer 3: Linear(256 -> 256) -> FiLM -> sin(30 * .)
head: Linear(256 -> 3) -> sigmoid = RGB
Parameter count still doesn't depend on resolution — --res 256 runs the same 919,427 weights as --res 64, same as v1 and v2.
Try it
git clone https://huggingface.co/bench-labs/PixelModel-v2
git clone https://huggingface.co/bench-labs/PixelModel-v3
python main.py "a red double decker bus" --out bus.png
python main.py "a beach with palm trees" --res 256
model.png is the canonical model for both; model.safetensors carries the same weights with the parameter breakdown in its header. Check out the PixelModel v2 and PixelModel v3 repositories, or the full text-to-image collection.
v2 proved width alone isn't the story. v3 is 919K parameters that earned their FID score honestly, against every prior version, on the same pipeline.