The 7-2026 generation: dual-mode benchmarks

All three Bench Labs tiers — Effortless, Easy, and Mid — have been remade from scratch. The new generation is called 7-2026, and the old 6-2026 datasets stay frozen forever, so every published score remains reproducible.

One schema, two scoring modes

The 6-2026 tiers had two incompatible formats: Effortless/Easy were generation-scored question/answer pairs, Mid was log-likelihood over target_scores. The 7-2026 schema unifies them: every item in every tier carries a gold answer (plus accepted aliases) and a set of scored choices. One run of the universal script now reports both metric families for each tier:

• generative — alias-aware exact_match, plus a hybrid_score whose scorer (strict / semantic / fuzzy) is chosen per item by a gen_scoring field baked into the data;
• loglikelihood — lm-eval style acc, acc_norm, soft_score, soft_score_norm, with per-choice log-probs (raw and per-byte) written to a per-item samples_*.jsonl.

Headline metrics stay tier-conventional — exact match on Effortless, hybrid on Easy, soft_score_norm on Mid — so leaderboard numbers remain comparable in spirit to before.

Math, remade and machine-verified

Every Math item in 7-2026 is programmatically verified. Arithmetic questions ship with a checked expression that must evaluate exactly to the gold answer using only numbers that appear in the question; sequence puzzles ship with an explicit rule in a tiny DSL, and the sequence plus next term are regenerated from that rule before the item is accepted. No more "the gold answer is wrong" issues in Math — the pipeline rejects them at the source. This also fixes a 6-2026 scoring bug where three categories silently fell through to fuzzy string matching and inflated scores.

Balanced, bigger, cleaner

• 300 items per tier (up from 143–240), every one of the 17 categories represented with 17–18 items in every tier;
• near-duplicate detection against all 6-2026 data and within the new generation itself;
• every non-math item was blind-answered by a second, different model — items the verifier could not defend were dropped;
• full per-item provenance: which model drafted it, when, from which prompt, and how it was verified.

How the data was made

Questions were drafted by free-tier LLMs used as blind single-shot text generators: no tools, no session memory, no knowledge of the benchmark, the taxonomy, or the schema — each call sees only "produce N quiz items of this one kind at this difficulty, as a JSON array". A driver script assembles the actual benchmark items, so the drafting models can never steer the format, and a separate model family cross-checks the answers blind.

The leaderboard filters by generation now

The leaderboard defaults to the latest generation, so new visitors always see current results. Older 6-2026 runs are one click away — nothing was invalidated, they're just not the default view anymore. The evaluation script keeps both generations runnable: --benchmarks effortless7,easy7,mid7 for the new tiers, effortless,easy,mid for the frozen ones.

Datasets: bench-effortless-7-2026 · bench-easy-7-2026 · bench-mid-7-2026