
SlopCodeBench (SCBench) is an MIT benchmark from a UW-Madison-led academic team that scores coding agents on 36 problems of 196 checkpoints where the agent repeatedly extends its own solution as the specification changes, measuring per-checkpoint correctness alongside the structural erosion and verbosity the kept code accumulates.

**The measured object is the agent living with its own earlier decisions: every checkpoint accepts any working implementation, while the erosion and verbosity metrics price the path the agent took to get there, which is the cost single-shot benchmarks never charge.**

## What it is

An MIT Python 3.12+ CLI (`slop-code run`, `slop-code eval`, `slop-code metrics judge`) that runs an agent inside per-problem Docker workspaces, from the SprocketLab organization.
Problems live in the sibling [scb-problems](https://github.com/gabeorlanski/scb-problems) repository (Apache-2.0, with a contributed-problem guide and acceptance criteria) and also publish as a Harbor dataset in the registry of the [Harbor](../harbor/index.md) runner, so the benchmark distributes without shipping the judging infrastructure.
Evaluation is black-box: the agent sees only a CLI or API contract, with no function signatures, module boundaries, or test suite, and verifiers normalize output formatting so arbitrary choices do not fail a checkpoint.
Two quality axes sit beside correctness: verbosity (a 45-criteria LLM-judge rubric plus handcrafted AST-Grep rules, looking for narration comments, unrequested features, and handrolled standard operations) and structural erosion (the share of functions over a cyclomatic complexity of 10, curated Ruff lint errors per line, and LOC and churn deltas between checkpoints).
The provenance is academic: ten authors led by Gabriel Orlanski with Aws Albarghouthi and Frederic Sala, supported by DARPA, the NSF, and Snorkel AI's Open Benchmarks Grant; the paper is arXiv 2603.24755 (v1 March 2026, v2 May 2026, CC BY 4.0), and results publish to the scbench.ai leaderboard (site label v1.0: 36 problems, 196 checkpoints, 22 models, as of 2026-11-22).

## Status

Active, young, with adoption beyond its makers.
The repository was created 2025-12-17, pushed 2026-11-19, and holds 324 stars and 57 forks as of 2026-11-22, on a release train of v0.2 (2026-01-24) and v0.3 (2026-04-21), with the website's v1.0 label sitting ahead of the published releases.
The paper's runs put 15 coding agents through the benchmark: no agent fully solves any problem end-to-end, the best agent passes 14.8% of checkpoints, structural erosion rises in 77% of trajectories and verbosity in 75.5%, and agent code reads 2.3x more verbose and 2.0x more eroded than 473 open-source Python repositories.
**The footprint is practitioner-grade and independent of the makers: HumanLayer runs the benchmark against every new frontier model and publishes the runs (a 405-point Hacker News thread on Opus 5 and a 309-point thread on the 10/2026 board), and the AI That Works podcast covered it three times, including #81 with the creator.**

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=SprocketLab/slop-code-bench&type=date&theme=dark&legend=top-left" />
  <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=SprocketLab/slop-code-bench&type=date&legend=top-left" />
  <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=SprocketLab/slop-code-bench&type=date&legend=top-left" />
</picture>

## Strengths

- It measures the failure mode single-shot benchmarks hide: an agent whose early architecture makes every later extension harder can pass every isolated checkpoint and still be the worst agent on the board.
- The correctness core is black-box and output-normalized, so a checkpoint cannot be passed by matching an internal structure the spec never asked for.
- Verbosity and erosion move independently, sometimes in opposite directions, so the two numbers expose trade-offs a single quality score would fold away.
- The problem base is decoupled from the runner: community problems flow through scb-problems with acceptance criteria, and the Harbor dataset registration means the tasks run inside tooling you may already use.

## Cautions

- The authors themselves frame the launch as an exploratory probe, the release train sits at v0.3 behind the website's v1.0 label, and the problems sibling has been quiet since 2026-10-07, so the task base is still small (36 problems) and moving.
- Half of the quality signal is an LLM-judge rubric, so the verbosity number inherits judge variance the way every model-judged metric in this category does; the lint and complexity halves are deterministic but narrower.
- The loudest adoption is one practitioner's campaign: HumanLayer's model shootouts are engaged, argumentative coverage, but they are still a single team running the board, and the leaderboard itself is maker-run.
- Problems are Python-first and CLI/API contracts, so the benchmark says nothing about frontend work, review quality, or non-coding agents.

## Pricing

Free, MIT; a run bills your agent's model tokens plus the verbosity judge's tokens, and Docker images build per agent version on first use.
The work is grant-supported (DARPA, NSF, Snorkel AI's Open Benchmarks Grant), so no paid tier exists.

## Compared to

- [SWE-bench](../swe-bench/index.md): the field's ground benchmark scores one patch against one frozen issue's tests; SlopCodeBench chains checkpoints onto the agent's own code, which is the measurement SWE-bench's single-shot setup cannot make.
- [Harbor](../harbor/index.md): the neutral runner that executes benchmark datasets; SlopCodeBench publishes its problems into Harbor's registry but ships its own runner and scoring stack, so you can run the tasks either way.
- [FrontierHarness Eval](../frontierharness-eval/index.md): that study holds the benchmark constant and varies the harness to price the harness choice; SlopCodeBench holds the harness roughly constant and varies the requirements, pricing how the code degrades as the spec grows.

## Bottom line

**Recommended for teams choosing a coding agent or model and asking what its output looks like to extend six months later, and for anyone quoting single-shot leaderboards who wants the counter-metric.**
Not for gating merges (an offline benchmark, like every benchmark column here) and not for judging non-coding agents.
My disagreeable claim: single-shot pass-rate leaderboards systematically overstate agent quality, because they never charge the agent for its own architectural decisions, and this benchmark's 14.8% best checkpoint pass rate is what the charged version looks like.

## Changes

- 2026-11-22 - Created from the coordinator's pre-screen, admitted as the category's iterative-extension coding-agent benchmark beside the single-shot ground benchmark.

## See also

- [SWE-bench](../swe-bench/index.md) - the single-shot ground benchmark this one argues against, and the board its critiques target
- [Harbor](../harbor/index.md) - the neutral runner whose registry distributes SCBench's problems as a dataset
- [FrontierHarness Eval](../frontierharness-eval/index.md) - the harness-axis benchmark study, the orthogonal comparison to SCBench's spec-axis one
- [MLR-Bench](../mlr-bench/index.md) - the other rubric-plus-judge benchmark in the category, pointed at research agents instead of coding agents

## References

- https://github.com/SprocketLab/slop-code-bench - repository: README, install, agent configs, the scb-problems and Harbor-dataset split (fetched 2026-11-22)
- https://api.github.com/repos/SprocketLab/slop-code-bench - stars, forks, dates, and the MIT license as of 2026-11-22
- https://arxiv.org/abs/2603.24755 - the paper: 36 problems, 196 checkpoints, the 15-agent runs, the 14.8% best pass rate, the 2.3x verbosity and 2.0x erosion comparisons against 473 human repositories
- https://gabeorlanski.github.io/posts/slop-code-bench/ - the launch post: design philosophy, both metric definitions, the initial 18-problem release, and the v0.1 exploratory-probe framing
- https://www.scbench.ai - the live leaderboard and the site's own v1.0 label (36 problems, 196 checkpoints, 22 models, fetched 2026-11-22)
- https://api.github.com/repos/gabeorlanski/scb-problems - the problems sibling: Apache-2.0, pushed 2026-10-07 (fetched 2026-11-22)
- https://api.github.com/repos/SprocketLab/slop-code-bench/releases - the release train: v0.2 (2026-01-24) and v0.3 (2026-04-21, new cursor_cli, pi, openhands agents)
- https://hn.algolia.com/api/v1/search?query=slop-code-bench&hitsPerPage=8 - the adoption footprint: the 405-point Opus 5 thread (2026-07-27) and the 309-point 10/2026 board thread (2026-10-15)
- https://registry.harborframework.com/datasets/gabeorlanski/slopcodebench/latest - the Harbor dataset registration (page shell fetched 2026-11-22; details load client-side)
- https://github.com/ai-that-works/ai-that-works - the AI That Works show repo: episodes #68 (2026-08-04), #79 (2026-10-27), and #81 (2026-11-17) with the creator (README fetched 2026-11-22)
