SWE-bench: What the Score Measures

SWE-bench measures how often an AI coding agent can resolve real GitHub issues: it gets a repository and an issue, writes a patch, and the patch counts only if the project's own tests then pass. The headline number is the percent of instances resolved. The swebench.com leaderboard we read on 31 October 2026 had entries up to 26 February 2026, so check the date on any score.

The short version

What the benchmark is

The original paper, "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (arXiv 2310.06770), built the test from pull requests that closed an issue in popular Python projects. Each task instance has four parts:

1. A snapshot of the repository from just before the fix.
2. The text of the GitHub issue, which is the only instruction the agent gets.
3. Tests that failed before the fix and pass after it (the ones the fix must repair).
4. Tests that passed before and must still pass (the ones the fix must not break).

The agent has to produce a patch. It does not see the tests that grade it. That is closer to real work than a short coding puzzle: the agent must find the right files in a large codebase, understand an issue written by a stranger, and change code without breaking something else.

The variants

The swebench.com home page lists the family. Instance counts and descriptions below are from that page.

Verified deserves a note. The site's news list says "SWE-bench x OpenAI = SWE-bench Verified" in August 2024. The point of the filtering was to remove tasks that were underspecified or whose tests rejected valid fixes, so a failure means the agent failed and not the task. When a vendor says "SWE-bench" without a suffix, they almost always mean Verified. Ask which one.

The team also lists newer projects on the same site: CodeClash (November 2025), which evaluates models as goal-oriented developers rather than task solvers, and ProgramBench (May 2026), which asks whether models can build meaningful software from scratch. Neither is SWE-bench, and we do not quote scores from them here.

How a run is graded

The harness is open source at github.com/SWE-bench/SWE-bench. The news list notes that it was made Docker-based in June 2024 so results are easier to reproduce. For each instance the harness:

• Builds a container with the repository at the base commit and its dependencies.
• Applies the agent's patch.
• Runs the specified tests.
• Marks the instance resolved only if every required test passes.

The agent does not have to be a model alone. It is a model plus a loop: the code that reads files, runs commands and decides when to stop. That loop is the "scaffold" or "harness," and it matters. The leaderboard rows carry both an agent name and a model. A single model can appear with different agents and very different scores.

How scores are reported

Every entry reports % Resolved, the share of the task instances that were solved. Entries also carry fields that readers tend to skip:

• Agent and organisation. For example mini-SWE-agent from the SWE-agent group, or a company's own product.
• Attempts. The tag data separates "Attempts - 1" from "Attempts - 2+". A run with several tries per task and a selection step is easier than a single try.
• Cost and step limits. The site has charts for resolved versus cost, resolved versus step limit and cumulative cost distribution. A run that spends ten times as much is not directly comparable to a cheap one.
• A "checked" mark. The site's key says which runs were "performed or directly checked by the SWE-bench team." Unchecked runs are submitted by the party that benefits from the number.
• Open-weights flag. Open-weights models are marked, so you can filter for them.

Published leaderboard figures

These come from the leaderboard data file in the swebench.com repository, read on 31 October 2026. We copied only the rows below, and we did not run any of these tests.

Bash Only view on Verified (mini-SWE-agent 2.0.0, one agent for all models), entries dated 17 February 2026

Other Verified entries with their own agents

Three things to notice. First, the same family of model scores about three points higher with a company's own agent (79.2) than inside the common mini-SWE-agent environment (76.8). That gap is the harness. Second, the cheapest runs are close to the top: MiniMax M2.5 sits level with Gemini 3 Flash at far lower reported cost in the same file, so cost belongs next to the score. Third, none of these are the models released in the autumn of 2026. The data file's newest entry we saw was dated 26 February 2026.

SWE-bench Multilingual, mini-SWE-agent 2.0.0a0, entries dated 13 February 2026

Vendor-reported SWE-bench figures on this site

Model makers publish their own numbers on their model cards. Our model pages record some of them, always labelled vendor-reported:

• NVIDIA's table for its Nemotron models reports 70.7 on SWE-Bench Verified for Ultra, and shows several rival open models higher in the same table. See Nemotron models.
• Mistral's card lists 77.6% on SWE-Bench Verified for Mistral Medium 3.5. See the best open source LLM for coding.

Compare either figure with the leaderboard rows above and you will see why we do not sort models by a mix of both. Different agents, different effort levels and different attempt counts sit behind each number. Anthropic's recent launch posts, for example, headline Terminal-Bench 4.0 rather than SWE-bench; see Terminal-Bench.

Limits and caveats

Python-heavy and public

The original tasks come from 12 Python repositories. A model can look excellent on them and struggle in a Go service or a PHP plugin. Multilingual (9 languages) exists to ease that, with a much smaller task count.

The tasks are public

Public tasks and fixes can end up in training data. Any benchmark with public answers carries this risk. The Terminal-Bench team's own integrity update shows what happens in practice: agents found solutions on the internet and the team had to rescore them. See how to read AI benchmarks.

Tests are a proxy

A patch that passes the tests is not always a patch a maintainer would merge. It may fix the symptom, ignore style or leave a worse design. SWE-bench grades the tests, not the review.

One shot versus a workflow

Real teams run an agent, read the diff, and ask for a change. The benchmark gives the agent one issue and measures the first result. It tells you little about how good the agent is at a conversation, or at working with your review rules.

Saturation

When top scores sit in the high seventies, the last few points are mostly the hard or flawed tasks. Small differences between two models are within the noise that different harness settings introduce.

Reading a SWE-bench claim, step by step

1. Find the variant. Verified, Lite, Multilingual or full.
2. Find the agent. Is it a common one such as mini-SWE-agent, or the vendor's own?
3. Find the effort and attempts. "High" reasoning and several attempts raise the score and the bill.
4. Look for a check mark. Did the SWE-bench team run it?
5. Look at the date. A score from February tells you little about a model released in September.
6. Look for cost. Resolved per dollar is a better working number than resolved alone.

What to do with the numbers

For an open-weights shortlist, the best open source LLM for coding collects vendor-reported coding scores with their sources. For the broader picture of coding evaluations, see Aider polyglot and the benchmark hub.

A small test you can run yourself

You do not need the full harness to learn something. Pick ten closed bugs from your own repository that each have a test, check out the commit before each fix, and give an agent the issue text only. Count how many it fixes with the tests passing. Record the agent, model, effort setting and cost. That is a SWE-bench-shaped test of your own code, and it is more relevant to you than any public number. Keep the method fixed when you compare two models, or you are measuring your setup.

Worked example: cost per resolved issue

Scores hide price, so here is the arithmetic. This is an example with invented round numbers, not a leaderboard figure. Suppose agent A resolves 76 of 100 tasks and spends 300 dollars in total, while agent B resolves 72 of 100 and spends 40 dollars.

Agent A wins the headline by four points. Agent B costs about a seventh as much per fix. If your team reviews every patch anyway, the cheaper agent can be the better buy, because the bottleneck is the reviewer. The swebench.com cost charts exist so you can do this sum with real rows. Our own model pages follow the same habit: see Claude pricing and LLM API prices for the per-token side.

Terms you will meet

• Instance. One task: a repository snapshot plus an issue and its tests.
• Resolved. The patch applies and all required tests pass.
• FAIL_TO_PASS tests. Tests that must flip from failing to passing.
• PASS_TO_PASS tests. Tests that must keep passing.
• Scaffold or harness. The loop of tools and prompts around the model.
• Pass@1. One attempt per task, no retries. A leaderboard tag of "Attempts - 1" is the closest match.
• Contamination. When test material leaks into training data and inflates a score.
• Saturation. When top scores bunch so close that the test stops separating models.

How SWE-bench fits with the other coding tests

No one of them tells you how an agent behaves on your repository. Used together they cover three different skills, and a model that is strong on all three is a safer bet than one that tops a single list.

Our take

SWE-bench is the best-known test of whether an agent can change real code and keep the tests green, and the Verified filtering made it fairer. It is still one Python-heavy test with public tasks, run through harnesses that move the number by points. We use it to see direction and rank within the same agent, never to pick a tool by itself. We have not run these tests, and every figure here is a published one with its source and date.

Related: how to read AI benchmarks, Terminal-Bench, Aider polyglot, AI coding agents and our methodology.