SlopCodeBench (SCBench) is an MIT benchmark from a UW-Madison-led academic team that scores coding agents on 36 problems of 196 checkpoints where the agent repeatedly extends its own solution as the specification changes, measuring per-checkpoint correctness alongside the structural erosion and verbosity the kept code accumulates.
The measured object is the agent living with its own earlier decisions: every checkpoint accepts any working implementation, while the erosion and verbosity metrics price the path the agent took to get there, which is the cost single-shot benchmarks never charge.
What it is #
An MIT Python 3.12+ CLI (slop-code run, slop-code eval, slop-code metrics judge) that runs an agent inside per-problem Docker workspaces, from the SprocketLab organization.
Problems live in the sibling scb-problems repository (Apache-2.0, with a contributed-problem guide and acceptance criteria) and also publish as a Harbor dataset in the registry of the Harbor runner, so the benchmark distributes without shipping the judging infrastructure.
Evaluation is black-box: the agent sees only a CLI or API contract, with no function signatures, module boundaries, or test suite, and verifiers normalize output formatting so arbitrary choices do not fail a checkpoint.
Two quality axes sit beside correctness: verbosity (a 45-criteria LLM-judge rubric plus handcrafted AST-Grep rules, looking for narration comments, unrequested features, and handrolled standard operations) and structural erosion (the share of functions over a cyclomatic complexity of 10, curated Ruff lint errors per line, and LOC and churn deltas between checkpoints).
The provenance is academic: ten authors led by Gabriel Orlanski with Aws Albarghouthi and Frederic Sala, supported by DARPA, the NSF, and Snorkel AI’s Open Benchmarks Grant; the paper is arXiv 2603.24755 (v1 March 2026, v2 May 2026, CC BY 4.0), and results publish to the scbench.ai leaderboard (site label v1.0: 36 problems, 196 checkpoints, 22 models, as of 2026-11-22).
Status #
Active, young, with adoption beyond its makers. The repository was created 2025-12-17, pushed 2026-11-19, and holds 324 stars and 57 forks as of 2026-11-22, on a release train of v0.2 (2026-01-24) and v0.3 (2026-04-21), with the website’s v1.0 label sitting ahead of the published releases. The paper’s runs put 15 coding agents through the benchmark: no agent fully solves any problem end-to-end, the best agent passes 14.8% of checkpoints, structural erosion rises in 77% of trajectories and verbosity in 75.5%, and agent code reads 2.3x more verbose and 2.0x more eroded than 473 open-source Python repositories. The footprint is practitioner-grade and independent of the makers: HumanLayer runs the benchmark against every new frontier model and publishes the runs (a 405-point Hacker News thread on Opus 5 and a 309-point thread on the 10/2026 board), and the AI That Works podcast covered it three times, including #81 with the creator.
Strengths #
- It measures the failure mode single-shot benchmarks hide: an agent whose early architecture makes every later extension harder can pass every isolated checkpoint and still be the worst agent on the board.
- The correctness core is black-box and output-normalized, so a checkpoint cannot be passed by matching an internal structure the spec never asked for.
- Verbosity and erosion move independently, sometimes in opposite directions, so the two numbers expose trade-offs a single quality score would fold away.
- The problem base is decoupled from the runner: community problems flow through scb-problems with acceptance criteria, and the Harbor dataset registration means the tasks run inside tooling you may already use.
Cautions #
- The authors themselves frame the launch as an exploratory probe, the release train sits at v0.3 behind the website’s v1.0 label, and the problems sibling has been quiet since 2026-10-07, so the task base is still small (36 problems) and moving.
- Half of the quality signal is an LLM-judge rubric, so the verbosity number inherits judge variance the way every model-judged metric in this category does; the lint and complexity halves are deterministic but narrower.
- The loudest adoption is one practitioner’s campaign: HumanLayer’s model shootouts are engaged, argumentative coverage, but they are still a single team running the board, and the leaderboard itself is maker-run.
- Problems are Python-first and CLI/API contracts, so the benchmark says nothing about frontend work, review quality, or non-coding agents.
Pricing #
Free, MIT; a run bills your agent’s model tokens plus the verbosity judge’s tokens, and Docker images build per agent version on first use. The work is grant-supported (DARPA, NSF, Snorkel AI’s Open Benchmarks Grant), so no paid tier exists.
Compared to #
- SWE-bench: the field’s ground benchmark scores one patch against one frozen issue’s tests; SlopCodeBench chains checkpoints onto the agent’s own code, which is the measurement SWE-bench’s single-shot setup cannot make.
- Harbor: the neutral runner that executes benchmark datasets; SlopCodeBench publishes its problems into Harbor’s registry but ships its own runner and scoring stack, so you can run the tasks either way.
- FrontierHarness Eval: that study holds the benchmark constant and varies the harness to price the harness choice; SlopCodeBench holds the harness roughly constant and varies the requirements, pricing how the code degrades as the spec grows.
Bottom line #
Recommended for teams choosing a coding agent or model and asking what its output looks like to extend six months later, and for anyone quoting single-shot leaderboards who wants the counter-metric. Not for gating merges (an offline benchmark, like every benchmark column here) and not for judging non-coding agents. My disagreeable claim: single-shot pass-rate leaderboards systematically overstate agent quality, because they never charge the agent for its own architectural decisions, and this benchmark’s 14.8% best checkpoint pass rate is what the charged version looks like.
Changes #
- 2026-11-22 - Created from the coordinator’s pre-screen, admitted as the category’s iterative-extension coding-agent benchmark beside the single-shot ground benchmark.
See also #
- SWE-bench - the single-shot ground benchmark this one argues against, and the board its critiques target
- Harbor - the neutral runner whose registry distributes SCBench’s problems as a dataset
- FrontierHarness Eval - the harness-axis benchmark study, the orthogonal comparison to SCBench’s spec-axis one
- MLR-Bench - the other rubric-plus-judge benchmark in the category, pointed at research agents instead of coding agents
References #
https://github.com/SprocketLab/slop-code-bench - repository: README, install, agent configs, the scb-problems and Harbor-dataset split (fetched 2026-11-22)
https://api.github.com/repos/SprocketLab/slop-code-bench - stars, forks, dates, and the MIT license as of 2026-11-22
https://arxiv.org/abs/2603.24755 - the paper: 36 problems, 196 checkpoints, the 15-agent runs, the 14.8% best pass rate, the 2.3x verbosity and 2.0x erosion comparisons against 473 human repositories
https://gabeorlanski.github.io/posts/slop-code-bench/ - the launch post: design philosophy, both metric definitions, the initial 18-problem release, and the v0.1 exploratory-probe framing
https://www.scbench.ai - the live leaderboard and the site’s own v1.0 label (36 problems, 196 checkpoints, 22 models, fetched 2026-11-22)
https://api.github.com/repos/gabeorlanski/scb-problems - the problems sibling: Apache-2.0, pushed 2026-10-07 (fetched 2026-11-22)
https://api.github.com/repos/SprocketLab/slop-code-bench/releases - the release train: v0.2 (2026-01-24) and v0.3 (2026-04-21, new cursor_cli, pi, openhands agents)
https://hn.algolia.com/api/v1/search?query=slop-code-bench&hitsPerPage=8 - the adoption footprint: the 405-point Opus 5 thread (2026-07-27) and the 309-point 10/2026 board thread (2026-10-15)
https://registry.harborframework.com/datasets/gabeorlanski/slopcodebench/latest - the Harbor dataset registration (page shell fetched 2026-11-22; details load client-side)
https://github.com/ai-that-works/ai-that-works - the AI That Works show repo: episodes #68 (2026-08-04), #79 (2026-10-27), and #81 (2026-11-17) with the creator (README fetched 2026-11-22)