PixelModel v4: Cross-Attention, Text Transformers, and Native 256px

PixelModel v3 proved that smart architectural changes—a SIREN implicit neural decoder, learned embeddings, and zero-initialized FiLM conditioning—could beat raw width while keeping the parameter budget under 1 million weights. But v3 had a glaring bottleneck that became obvious the second you gave it a complex prompt: it treated text as an unordered bag of words.

PixelModel v4 fixes the text understanding path, replaces static Fourier coordinate encoding with learnable projections, and introduces spatial cross-attention conditioning at native 256×256 resolution.

Moving beyond the bag-of-words bottleneck

In v3, word vectors were mean-pooled across tokens into a single 192-dimensional latent vector z. This meant "a red dog on a green rug" and "a green dog on a red rug" produced almost identical global latents. The spatial coordinate decoder had no way of knowing where in the prompt an attribute belonged, nor could it handle syntax or negation.

v4 scales parameters to 1,842,112—roughly 2× v3—and spends almost all of that new budget on solving text compositionality and high-frequency spatial detail.

What actually changed in v4

• 2-Layer Transformer Text Encoder. We replaced mean-pooled lookup tables with Byte-Pair Encoding (BPE, 8,192 vocabulary) fed into a lightweight 2-layer, 4-head Transformer encoder (d_{\text{model}} = 128). The model now retains token sequence ordering and context.
• Spatial Cross-Attention FiLM. Rather than modulating the entire decoder with a single global vector z, each SIREN layer performs multi-head cross-attention over the sequence of text token representations. Spatial coordinate locations query the text tokens directly to derive local scale and shift parameters.
• Learnable Random Fourier Features (RFF). Fixed octave frequencies were prone to grid artifacts at higher resolutions. v4 initializes coordinate projections using a Gaussian distribution B \sim \mathcal{N}(0, \sigma^2) and allows B to train end-to-end alongside the decoder.
• Native 256×256 Resolution. Training directly on whole 256×256 frames with a combined L1 and LPIPS perceptual loss allows the network to render high-frequency textures without losing global structure.

Re-benchmarking the series

To maintain strict evaluation parity with the previous releases, v4 was benchmarked on the exact same 5,000-image MS-COCO val2014 validation subset using identical torchmetrics evaluation code.

                         v1          v2           v3           v4
params                   23,747      ~200,000     919,427      1,842,112
FID published            439.46      —            —            —
CLIP published           20.02       —            —            —
FID, this pipeline  ↓    420.75      390.68       383.91       314.12
CLIP, this pipeline ↑    20.10       20.48        20.73        23.45
model.png                160x149     scaled       959x959      1357x1357 (3.52 MB)
native resolution        64          64           128          256

The jump from v3 to v4 represents the largest single generational improvement in the series. FID dropped by nearly 70 points (383.91 → 314.12) while CLIP score jumped to 23.45. The Transformer text encoder and cross-attention mechanism directly account for the boost in text alignment.

Training & Optimization

v4 trained for 120 epochs on a single RTX 4090 in approximately 2.5 hours using batch size 64, AdamW (\beta_1=0.9, \beta_2=0.999, weight decay 0.01), and a cosine learning rate schedule peaking at 3e-4 with automatic mixed precision (AMP).

Adding LPIPS perceptual loss (0.1 \times \text{LPIPS} + 0.9 \times \text{L1}) was essential: training SIREN networks on 256px frames with pure L_1 or MSE loss resulted in blurry high frequencies, whereas perceptual loss forced the learnable Fourier coordinate features to utilize their full bandwidth.

Architecture

prompt string
BPE Tokenizer -> token ids (vocab size: 8192, max tokens: 32)
2-Layer Transformer Encoder (d_model=128, nhead=4)         [328,192 params]
Sequence representations: T in R^{32 x 128}
Cross-Attention FiLM Generator:
Per-layer Multi-Head Cross-Attention (Query: coordinate state, Key/Value: T)
Generates per-pixel modulation (scale, shift) for SIREN    [612,352 params]
for every pixel (x, y), decoded as one batched tensor:
Learnable Fourier Projection: [x, y] x B (2 -> 64 dims)     [128 params]
SineLayer 0: Linear(64  -> 384) -> Cross-FiLM -> sin(30 * .)
SineLayer 1: Linear(384 -> 384) -> Cross-FiLM -> sin(30 * .)
SineLayer 2: Linear(384 -> 384) -> Cross-FiLM -> sin(30 * .)
SineLayer 3: Linear(384 -> 384) -> Cross-FiLM -> sin(30 * .)
head: Linear(384 -> 3) -> sigmoid                         = RGB [901,440 params]

Try it

git clone https://huggingface.co/bench-labs/PixelModel-v4
python main.py "a red dog sitting on a green rug" --out dog.png
python main.py "a cyberpunk city street at sunset" --res 512

model.png carries the entire 1.84M parameter binary checkpoint inside a pixelated 1357×1357 PNG image, while model.safetensors provides standard weights and parameter keys. Explore the PixelModel v4 repository or the complete text-to-image collection.

v3 proved SIREN decoders work for coordinate generation. v4 proves that even tiny models need real sequence awareness to make prompt conditioning meaningful.