One Script, Every Benchmark
Until today, every Bench Labs benchmark had its own eval files — a notebook here, an lm-eval config there, a custom scorer somewhere else. It worked, but reproducing a leaderboard entry meant figuring out which pipeline produced it. That's over.
Download, run, done
curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py
pip install torch transformers
python script.py --model your/modelThat's the whole workflow. One script.py, hosted in the leaderboard Space itself, runs the full suite:
- bench-effortless-6-2026 — normalized exact-match.
- bench-easy-6-2026 — the hybrid category-aware scorer (strict / flexible / semantic), same routing as the original evaluator. Plain exact-match is reported alongside.
- bench-mid-6-2026 — lm-eval-style loglikelihood over
target_scores:acc,acc_norm,soft_score,soft_score_norm. - bench-AGI — auto-skipped while the scoring pipeline is reworked; it will slot in when ready.
Full category breakdown, not one number
Every run reports overall, per-category, and per-subcategory scores, with macro averages and stderr:
=== bench-mid-6-2026 (tier 3) ===
headline [soft_score_norm]: 0.6103 (n=155, stderr=0.0388)
also: acc=0.5806 acc_norm=0.6000 soft_score=0.5871
category n soft_score_norm
Math 28 0.675
- arithmetic 8 0.625
- pattern 10 0.800
- reasoning 10 0.500
...Straight into the leaderboard
Besides results.json and per-sample CSVs, the script writes leaderboard.json — an entry already in the exact shape of the leaderboard's models.json, category breakdown included. Submitting is now: run the script, paste the entry, open a PR on the Space. Same scoring for everyone, reproducible with one command.
Options worth knowing: --benchmarks effortless,easy,mid to pick tiers, --limit 10 for a smoke test, --device / --dtype / --batch-size for hardware, --no-chat-template to force plain QA prompting. --help has the rest.
Bench Labs · Simple, Reliable, Open sourced