June 2026 · Benchmark

bench-mid-6-2026 — Qwen/Qwen2.5-0.5B

Evaluation of Qwen/Qwen2.5-0.5B on bench-labs/bench-mid-6-2026 using lm-eval multiple-choice loglikelihood scoring (CPU, bf16).

Headline Results

Metric	Value	StdErr
acc	0.5806	0.0398
acc_norm	0.6000	0.0395
soft_score	0.5871	0.0394
soft_score_norm	0.6103	0.0388

Per-Category Breakdown

Category	N	acc	acc_norm	soft_score	soft_score_norm
Commonsense-causality	5	1.000	1.000	1.000	1.000
Commonsense-reasoning	10	0.700	0.600	0.710	0.620
Commonsense-simulation	10	0.300	0.500	0.300	0.500
Knowledge-basic	8	1.000	1.000	1.000	1.000
Knowledge-definitions	10	0.700	0.900	0.700	0.900
Language-comprehension	10	0.600	0.800	0.600	0.800
Language-structure	10	0.000	0.100	0.000	0.100
Language-transformation	10	0.500	0.600	0.590	0.640
Logic-consistency	5	0.200	0.200	0.200	0.200
Logic-deduction	10	0.500	0.500	0.500	0.500
Logic-pattern	10	0.400	0.400	0.400	0.400
Math-arithmetic	8	0.625	0.625	0.625	0.625
Math-pattern	10	0.800	0.800	0.800	0.800
Math-reasoning	10	0.600	0.500	0.600	0.500
Pattern-generation	9	0.778	0.667	0.778	0.667
Pattern-matching	10	1.000	0.800	1.000	0.900
Pattern-recognition	10	0.300	0.300	0.300	0.300

Artifacts

task: bench_mid_6_2026
dataset_path: json
dataset_name: null
dataset_kwargs:
  data_files:
    test: /home/user/eval.jsonl
output_type: multiple_choice
training_split: null
validation_split: null
test_split: test
num_fewshot: 0
doc_to_text: "Question: {{input}}\nAnswer:"
doc_to_choice: !function utils.doc_to_choice
doc_to_target: !function utils.doc_to_target
process_results: !function utils.process_results
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: acc_norm
    aggregation: mean
    higher_is_better: true
  - metric: soft_score
    aggregation: mean
    higher_is_better: true
  - metric: soft_score_norm
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1.0

Evaluation run: lm-eval 0.4.12 · transformers 5.12.1 · device: CPU · batch size: 8 · samples: 155