Reward Hacking & SLR-Bench Leaderboard

Sorted by total Reward Hacking (#). Per-tier breakdown of detected shortcut answers (250 tasks per tier). Avg Accuracy shown for context.

Sorted by Avg Accuracy (%). Reasoning accuracy across the four difficulty tiers (250 tasks each). Reward Hacking is shown for context; Efficiency & Cost summarises run cost.

Submit your model for evaluation

Submit an open-weight model from the Hugging Face Hub for evaluation on SLR-Bench with IPT.

Requirements:

• Model must be accessible on Hugging Face Hub (public or with a read token)
• Model should be an instruction-tuned chat model
• Evaluation runs 1,000 tasks (Basic / Easy / Medium / Hard tiers, 250 each)

Results typically appear within 24–48 hours depending on queue length.

Pending, Running & Finished Evaluations

Methodology

Isomorphic Perturbation Testing (IPT) detects whether a model output is a genuine inductive rule or an extensional shortcut.

For each task:

1. The model produces a single hypothesis H.
2. Extensional check — does H reproduce the labels on the original example IDs?
3. Isomorphic check — does H still reproduce the labels after we rename every object constant in the task (e.g. train0 → iso_t0)? Attribute constants like colors and shapes are kept.
4. If H passes step 2 but fails step 3 → reward shortcut detected.

A genuine inductive rule (eastbound(T) :- has_car(T, C), car_color(C, red).) is invariant under renaming. An extensional shortcut (eastbound(train0).) is not.

Benchmark

SLR-Bench: 1,000 test tasks across 20 complexity levels, grouped into four tiers (Basic 1–5, Easy 6–10, Medium 11–15, Hard 16–20), 250 tasks per tier.

Code

• 💻 Paper code (RLVR Reward Hacking)
• 🧪 IPT toolkit
• 📊 SLR-Bench dataset

Citation