Aider Polyglot: Code Editing in Six Languages

The Aider polyglot benchmark gives a language model 225 hard Exercism exercises across C++, Go, Java, JavaScript, Python and Rust and checks its edits with unit tests. The score is the percent solved within two attempts. The leaderboard data we read on 31 October 2026 ended at 3 October 2025, with gpt-5 (high) on top at 88.0%.

The short version

Why it exists

Aider is an open-source terminal coding tool, covered on our Aider page. Its team ran a code editing benchmark on Python exercises. By December 2024 that test had stopped separating models. The polyglot announcement says the original benchmark was saturating as top scores passed 80%. The leading score of 84.2% meant solving 112 of 133 exercises, leaving just 21, and new champions were advancing the record by one or two problems. It also says many of the 133 Python exercises were so easy that models as old as GPT 3.5 Turbo solved half of them, which inflates scores without telling you which models are better.

The fix was a harder, wider set. The polyglot benchmark uses the most difficult 225 of the 697 exercises Exercism offers for C++, Go, Java, JavaScript, Python and Rust. Exercism is a public set of practice problems with tests. The announcement frames the new test as one where the same editing skill is measured over more languages and fewer easy problems.

How a run works

For each of the 225 exercises, Aider is given the task and the starter files. The model proposes edits in an edit format that Aider can parse. Aider applies the edits and runs the exercise's unit tests. If the tests fail, the model gets the test output and a second try. The leaderboard keeps both numbers:

Two lessons from this. First, the headline mixes two attempts, so a model that is good at fixing its own mistakes from test output gains. Second, the edit format matters: a model that cannot follow the diff format loses points for formatting and not for coding.

Published leaderboard figures

The numbers below come from the polyglot leaderboard data file in the Aider repository and the leaderboard page, read on 31 October 2026. They are what the project published. We did not run them.

The page for the top row shows the details: gpt-5 (high) solved 198 of 225 by the second attempt and 117 on the first (52.0% pass rate 1), with Aider version 0.86.2.dev, 194 seconds per case, and about 2.7 million prompt tokens and 2.6 million completion tokens.

What to take from it:

• Cost varies more than score. o3-pro (high) at 84.9% cost $146.32, while gpt-5 (medium) at 86.7% cost $17.69.
• Effort settings are separate rows. gpt-5 appears at high, medium and low, from 88.0% down to 81.3%, with cost roughly halving at each step down.
• Format failures are real. The top row got a well-formed reply 91.6% of the time, which means about one reply in twelve broke the edit format.
• The list is old. The newest row was dated 3 October 2025. Any model released after that is not on it, and that includes the Claude 5.5, GPT-6, Gemini 4 and Qwen3.8 generations described elsewhere on this site.

Why a stale board still has uses

The data tells you how the test works and which published rows are comparable. It can anchor a conversation about cost against accuracy. It cannot rank a model released in 2026. For current coding evidence use vendor-reported tables with their footnotes, such as the ones on Claude Opus, and public leaderboards that carry recent dates, such as SWE-bench and Terminal-Bench.

Limits and caveats

Exercises are small

Exercism problems are self-contained functions and small programs. They test whether a model can write correct code and edit a few files. They do not test finding the right file in a large repository, which is what SWE-bench does.

Public problems

Exercism exercises and their solutions are public. A model might have seen them. The team chose the hardest subset partly to reduce the chance that easy memorised answers decide the result, but the risk of leakage remains for any public set.

Model plus tool

The result is a model run through Aider's prompts and edit formats. A model tuned for another tool's format can look worse here than it would elsewhere. The correct-format column helps you see when that is happening.

Two attempts

The second attempt lets the model respond to failing tests. That mirrors how people work, but it makes the number higher than a single-shot score. Compare like with like: first attempts to first attempts, or headline to headline.

Cost is a snapshot

The cost column reflects prices on the run date. Prices have moved since; see LLM API prices for current per-token figures.

Arithmetic on the published rows: cost per solved exercise

The leaderboard gives cost and the number of exercises solved, so you can divide. This is our arithmetic on the published figures, not a number Aider publishes. Solved counts come from the percent correct times 225.

Read across and the pattern is clear. Dropping gpt-5 from high to medium lost three exercises and saved about 11 dollars on the run. Dropping to low lost another twelve and saved about 7 more. o3-pro bought eight fewer solved exercises than gpt-5 (high) for five times the money. A team running thousands of edits a day would price these differences very differently from a person running a few a week. The same logic applies to any model you test: record cost next to the pass count.

Edit formats in plain words

Aider asks the model to describe changes in a text format that the tool can apply to files. The leaderboard lists the format for each run.

Whole-file replies are simple but use many output tokens. Diff replies are compact but need care: if the search text does not match the file exactly the edit fails, and that failure counts against the "well formed" percentage. A model with a 92% well-formed rate loses roughly one reply in twelve to formatting alone. This is why the column exists, and why a model that is brilliant at reasoning can still post a middling score here. When you pick a model for Aider, the format guidance on the Aider page matters as much as the rank.

Running it yourself

The Aider repository includes the benchmark harness. A full run is 225 exercises per model, which costs real money: the published runs above range from about 10 dollars to over 140 dollars. Before you start, work out your budget from the cost column for a similar model, use a test run on a small subset first, and run the harness in a container, because it executes model-written code.

If you only want a feel for a model, run Aider itself on three of your own small tasks with the model set through its --model option. The leaderboard's own command column shows the form, for example aider --model openai/gpt-5. Check how many replies break the edit format and how many tasks pass after the second attempt. Those two counts mirror the benchmark's main signals.

Aider polyglot in an agent decision

Polyglot is a model benchmark, delivered through one open-source tool. If you use Aider daily it is directly useful. If you use Claude Code, Cursor or Codex, the prompts and edit mechanism differ, so the ranking does not transfer cleanly. A model that is strong at Aider's diff format is probably strong at editing code in general, but "probably" is the right word. For agent choices start from AI coding agents and choose an AI coding agent. For a local model on your own hardware, polyglot-style evidence matters for small models, where formatting errors are common; see Ollama and running an LLM locally.

How to read it

1. Check the date before the score.
2. Compare rows with the same edit format and similar effort settings.
3. Look at cost and seconds per case next to percent correct.
4. Check the well-formed percentage to see whether formatting is eating points.
5. Treat differences of a point or two as noise.

What to do with it

A quick way to repeat the idea

You can mimic the structure for your own code in an afternoon. Pick fifteen small functions in your language, each with a unit test. Remove the body of each, give the model the stub and the test file only, and let it try twice, showing it the test output after the first failure. Score first-try and second-try passes, and note the cost. Do it for two models and you have a polyglot-shaped test on your own stack. If you work in PHP, Ruby or another language not in the set, that is more useful than any published number.

Checklist before you quote a polyglot number

• The row has a date, and you have said what it is.
• You know the effort setting, because gpt-5 alone spans seven points across its three rows.
• You quote the cost with the score, or say that you did not.
• You say "after two attempts" if the figure is pass rate 2.
• You have not mixed polyglot with the older Python-only Aider benchmark, which stopped separating models.
• You describe the result as published by Aider and not as something you measured.

Our take

Aider polyglot was a sensible fix to a saturated test: harder problems, more languages, an honest second attempt and cost on the board. Its weakness today is age. The published rows stop in October 2025, so we use it to explain method and as a cost-against-accuracy example, not to rank current models. We have not run it. See the benchmark hub, how to read AI benchmarks and our methodology.