<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>swebench on tomrochette.com</title>
    <link>https://tomrochette.com/tags/swebench/</link>
    <description>Recent content in swebench on tomrochette.com</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en</language>
    <managingEditor>tom@tomrochette.com (Tom Rochette)</managingEditor>
    <webMaster>tom@tomrochette.com (Tom Rochette)</webMaster>
    <copyright>© 2026 Tom Rochette</copyright>
    <lastBuildDate>Thu, 29 Oct 2026 03:16:47 -0400</lastBuildDate><atom:link href="https://tomrochette.com/tags/swebench/index.xml" rel="self" type="application/rss+xml" />
    
    <item>
      <title>SWE-agent</title>
      <link>https://tomrochette.com/agents/harnesses/swe-agent/</link>
      <pubDate>Wed, 28 Oct 2026 00:00:00 +0000</pubDate>
      <author>tom@tomrochette.com (Tom Rochette)</author>
      <guid>https://tomrochette.com/agents/harnesses/swe-agent/</guid>
      <category>research-note</category><category>agent-curated</category><category>fully-ai-generated</category><category>llm=glm-5.3-flash</category><category>swebench</category><category>research</category><category>terminal</category>
      <description>&lt;p&gt;SWE-agent is the Princeton and Stanford research harness that introduced the agent-computer interface (ACI) idea, carried the model into a real repository through a constrained shell tool set, and became the reference agent of the early SWE-bench era.&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;What it is&#xA;    &lt;div id=&#34;what-it-is&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#what-it-is&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;p&gt;An open-source (MIT) Python terminal harness from the SWE-bench team, published with a NeurIPS 2024 paper arguing that interface design, not just model quality, decides agent performance.&#xA;The agent works inside a Docker-containerized copy of a repository through a yaml-governed set of tool bundles (an edit-lint flow, search, and a submit gate), and every run leaves a complete &lt;code&gt;.traj&lt;/code&gt; trajectory file.&#xA;It is built and maintained by researchers from Princeton University and Stanford University, sits beside their SWE-bench, SWE-ReX, and SWE-smith projects, and carries an EnIGMA mode for offensive-security capture-the-flag work pinned to the older v0.7 line.&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Status&#xA;    &lt;div id=&#34;status&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#status&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;p&gt;Maintenance-only by its own declaration, and still the category&amp;rsquo;s most cited ancestor: 20,663 stars on a repository created April 2, 2024, with the default branch pushed October 26, 2026 (GitHub API, as of 2026-10-28).&#xA;The newest tagged release is v1.1.0 (May 22, 2025, a training-trajectories drop), and no release has followed in seventeen months, while the docs site and README now carry a standing banner: most development effort moved to mini-swe-agent, which &amp;ldquo;has superseded SWE-agent&amp;rdquo;, and &amp;ldquo;SWE-agent is now in maintenance-only mode&amp;rdquo;.&#xA;The April 2, 2024 launch thread drew 307 points and 143 comments on Hacker News, and the paper&amp;rsquo;s 12.5 percent SWE-bench pass@1 was the open-source state of the art at publication.&lt;/p&gt;&#xA;&lt;picture&gt;&#xA;  &lt;source media=&#34;(prefers-color-scheme: dark)&#34; srcset=&#34;https://api.star-history.com/chart?repos=SWE-agent/SWE-agent&amp;type=date&amp;theme=dark&amp;legend=top-left&#34; /&gt;&#xA;  &lt;source media=&#34;(prefers-color-scheme: light)&#34; srcset=&#34;https://api.star-history.com/chart?repos=SWE-agent/SWE-agent&amp;type=date&amp;legend=top-left&#34; /&gt;&#xA;  &lt;img alt=&#34;Star History Chart&#34; src=&#34;https://api.star-history.com/chart?repos=SWE-agent/SWE-agent&amp;type=date&amp;legend=top-left&#34; /&gt;&#xA;&lt;/picture&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Strengths&#xA;    &lt;div id=&#34;strengths&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#strengths&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The ACI thesis is the field&amp;rsquo;s clearest statement that harness design is a first-class variable, and the paper&amp;rsquo;s ablations still anchor the argument.&lt;/li&gt;&#xA;&lt;li&gt;Everything is yaml: model, templates, history processors, demonstration trajectories, and tool bundles, which makes run configurations reproducible and shareable.&lt;/li&gt;&#xA;&lt;li&gt;Model routing through litellm covers essentially every provider including local Ollama-style endpoints, with a custom cost registry for unpriced models.&lt;/li&gt;&#xA;&lt;li&gt;Batch mode (&lt;code&gt;run-batch&lt;/code&gt;) with per-thread API-key rotation was built for benchmark evaluation, and &lt;code&gt;.traj&lt;/code&gt; outputs make every run auditable.&lt;/li&gt;&#xA;&lt;li&gt;Prompt-caching support for Claude via a &lt;code&gt;cache_control&lt;/code&gt; history processor shows unusual attention to inference cost for an academic tool.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Cautions&#xA;    &lt;div id=&#34;cautions&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#cautions&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The project recommends its own replacement: the supersession banner names mini-swe-agent as the going-forward choice, and maintenance-only means no release since May 2025.&lt;/li&gt;&#xA;&lt;li&gt;The SWE-bench numbers it rode on are themselves contested: the SWE-Bench+ analysis found solution leakage in 32.67 percent of SWE-Agent+GPT-4&amp;rsquo;s successful patches and weak tests in another 31.08 percent, dropping its measured resolution rate from 12.47 to 3.97 percent.&lt;/li&gt;&#xA;&lt;li&gt;The 1.0 rewrite broke the 0.7 configuration surface, and EnIGMA (the security mode) still documents the old version as its supported line.&lt;/li&gt;&#xA;&lt;li&gt;No subagents, no MCP, no editor integration: the loop is one model over one shell, by design.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Pricing&#xA;    &lt;div id=&#34;pricing&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#pricing&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;p&gt;Free and open source under MIT; pricing does not apply.&#xA;The cost of running it is your model API spend, which the litellm registry tracks per run.&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Compared to&#xA;    &lt;div id=&#34;compared-to&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#compared-to&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;p&gt;Mini-SWE-agent is the same team&amp;rsquo;s successor, a deliberately tiny loop with matching SWE-bench performance, and the project&amp;rsquo;s own recommendation for anything new.&#xA;OpenHands is the platform-scale research agent, with a self-hostable UI and cloud, where SWE-agent stays a scriptable command-line research instrument.&#xA;Claude Code and the other commercial members of this category are production harnesses; SWE-agent is the instrument the category&amp;rsquo;s evaluation culture grew up around.&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Bottom line&#xA;    &lt;div id=&#34;bottom-line&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#bottom-line&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;p&gt;Recommended for research on agent-harness design and for reproducible SWE-bench batch runs where the &lt;code&gt;.traj&lt;/code&gt; records and yaml configs are the point.&#xA;Not for daily production coding, where the project itself points you to mini-swe-agent or a commercial harness.&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Changes&#xA;    &lt;div id=&#34;changes&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#changes&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;2026-10-28 - Created from the cross-category handoff resolving SWE-agent/SWE-agent into the harnesses category.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;See also&#xA;    &lt;div id=&#34;see-also&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#see-also&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/harnesses/mini-swe-agent/&#34; &gt;Mini-SWE-agent&lt;/a&gt; - the successor this project itself recommends, from the same team.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/harnesses/openhands/&#34; &gt;OpenHands&lt;/a&gt; - the platform-scale research agent that grew out of the same benchmark wave.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/harnesses/aider/&#34; &gt;aider&lt;/a&gt; - the other pre-2025 open-source harness still in the corpus, stalled for different reasons.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/harnesses/harness-feature-matrix/&#34; &gt;Harness Feature Matrix&lt;/a&gt; - the category-wide comparison this note feeds a column into.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;References&#xA;    &lt;div id=&#34;references&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#references&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://api.github.com/repos/SWE-agent/SWE-agent&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=api.github.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://api.github.com/repos/SWE-agent/SWE-agent&lt;/a&gt; - repository metadata: stars, license, push date, and creation date (as of 2026-10-28)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://github.com/SWE-agent/SWE-agent&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=github.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://github.com/SWE-agent/SWE-agent&lt;/a&gt; - README: the supersession banner, feature claims, EnIGMA, and the sibling projects&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://swe-agent.com/latest/&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=swe-agent.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://swe-agent.com/latest/&lt;/a&gt; - the documentation home: the maintenance-only banner, news, and doc structure&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://swe-agent.com/latest/config/models/&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=swe-agent.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://swe-agent.com/latest/config/models/&lt;/a&gt; - litellm model routing, local models, Claude prompt caching, and key rotation&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://swe-agent.com/latest/faq/&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=swe-agent.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://swe-agent.com/latest/faq/&lt;/a&gt; - local model support, custom tools, multimodal configuration, and &lt;code&gt;.traj&lt;/code&gt; outputs&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://swe-agent.com/latest/usage/adding_custom_tools/&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=swe-agent.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://swe-agent.com/latest/usage/adding_custom_tools/&lt;/a&gt; - the tool-bundle structure behind the tools row&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://arxiv.org/abs/2405.15793&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=arxiv.org&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://arxiv.org/abs/2405.15793&lt;/a&gt; - the NeurIPS 2024 paper: the ACI thesis and the 12.5 percent SWE-bench result&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://arxiv.org/abs/2410.06992&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=arxiv.org&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://arxiv.org/abs/2410.06992&lt;/a&gt; - SWE-Bench+: the critical analysis of SWE-Agent+GPT-4 resolution rates, leakage, and weak tests&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://hn.algolia.com/api/v1/items/39907468&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=hn.algolia.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://hn.algolia.com/api/v1/items/39907468&lt;/a&gt; - the 307-point April 2, 2024 launch thread&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</description>
      
    </item>
    
  </channel>
</rss>
