Skip to content
Weights Not Walls
Menu

Aider polyglot

Reported

Coding exercises in several programming languages, where the model must edit code so that the exercise's tests pass.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Link
https://aider.chat/docs/leaderboards/#polyglot-leaderboard
Score shown
Epoch's column "Percent correct", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
Results
72 in all: 23 for open-weight models, 48 for closed ones, 1 where Epoch does not say. 19 linked to a model page here.

Best open-weight result: deepseek-reasoner, 74.2%, number 13 of 72 results.

Aider polyglot results
#ModelWeightsSettingScoreDateSource
1gpt-5-2025-08-07_highOpenAIclosedeffort high; Edit format: diff88.0%23 Aug 2025aider.chat
2gpt-5-2025-08-07_mediumOpenAIclosedeffort medium; Edit format: diff86.7%25 Aug 2025aider.chat
3o3-pro-2025-06-10_highOpenAIclosedeffort high; Edit format: diff84.9%28 Jun 2025aider.chat
4gemini-2.5-pro-preview-06-05_32KGoogle DeepMindclosedEdit format: diff-fenced83.1%6 Jun 2025aider.chat
5gpt-5-2025-08-07_lowOpenAIclosedeffort low; Edit format: diff81.3%25 Aug 2025aider.chat
6o3-2025-04-16_highOpenAIclosedeffort high; Edit format: diff81.3%25 Jun 2025aider.chat
7grok-4-0709xAIclosedEdit format: diff79.6%not datedAider LLM Leaderboards
8grok-4-0709_highxAIclosedeffort high; Edit format: diff79.6%11 Jul 2025aider.chat
9gemini-2.5-pro-preview-06-05Google DeepMindclosedEdit format: diff-fenced79.1%6 Jun 2025aider.chat
10gemini-2.5-pro-preview-05-06Google DeepMindclosedEdit format: diff-fenced76.9%7 May 2025aider.chat
11o3-2025-04-16_mediumOpenAIclosedeffort medium; Edit format: diff76.9%not datedAider LLM Leaderboards
12o3-2025-04-16_unknownOpenAIclosedeffort unknown; Edit format: diff76.9%25 Jun 2025aider.chat
13deepseek-reasonerDeepSeekno page matchopenEdit format: diff74.2%3 Oct 2025aider.chat
14deepseek-ai/DeepSeek-V3.2-ExpEpoch: DeepSeek-V3.2-Exp_thinking, DeepSeekopeneffort thinking; Edit format: diff74.2%not datedAider LLM Leaderboards
15gemini-2.5-pro-exp-03-25Google DeepMindclosedEdit format: diff-fenced72.9%not datedAider LLM Leaderboards
16gemini-2.5-pro-preview-03-25Google DeepMindclosedEdit format: diff-fenced72.9%12 Apr 2025aider.chat
17claude-opus-4-20250514_32KAnthropicclosedEdit format: diff72.0%25 May 2025aider.chat
18o4-mini-2025-04-16_highOpenAIclosedeffort high; Edit format: diff72.0%16 Apr 2025aider.chat
19deepseek-ai/DeepSeek-R1-0528Epoch: DeepSeek-R1-0528, DeepSeekopenEdit format: diff71.4%6 Jun 2025aider.chat
20claude-opus-4-20250514AnthropicclosedEdit format: diff70.7%25 May 2025aider.chat
21deepseek-chatDeepSeekno page matchopenEdit format: diff70.2%3 Oct 2025aider.chat
22deepseek-ai/DeepSeek-V3.2-ExpEpoch: DeepSeek-V3.2-Exp, DeepSeekopenEdit format: diff70.2%not datedAider LLM Leaderboards
23claude-3-7-sonnet-20250219_32KAnthropicclosedEdit format: diff64.9%24 Feb 2025aider.chat
24o1-2024-12-17_highOpenAIclosedeffort high; Edit format: diff61.7%21 Dec 2024aider.chat
25claude-sonnet-4-20250514_32KAnthropicclosedEdit format: diff61.3%24 May 2025aider.chat
26claude-3-7-sonnet-20250219AnthropicclosedEdit format: diff60.4%24 Feb 2025aider.chat
27o3-mini-2025-01-31_highOpenAIclosedeffort high; Edit format: diff60.4%31 Jan 2025aider.chat
28Qwen/Qwen3-235B-A22BEpoch: Qwen3-235B-A22B, AlibabaopenEdit format: diff59.6%not datedAider LLM Leaderboards
29Qwen/Qwen3-235B-A22B-Instruct-2507Epoch: Qwen3-235B-A22B-Instruct-2507, AlibabaopenEdit format: diff59.6%9 May 2025aider.chat
30moonshotai/Kimi-K2-InstructEpoch: Kimi-K2-Instruct, MoonshotopenEdit format: diff59.1%not datedAider LLM Leaderboards
31moonshotai/kimi-k2-0905Moonshotno page matchopenEdit format: diff59.1%17 Jul 2025aider.chat
32deepseek-ai/DeepSeek-R1Epoch: DeepSeek-R1, DeepSeekopenEdit format: diff56.9%20 Jan 2025aider.chat
33claude-sonnet-4-20250514AnthropicclosedEdit format: diff56.4%24 May 2025aider.chat
34deepseek-ai/DeepSeek-V3-0324Epoch: DeepSeek-V3-0324, DeepSeekopenEdit format: diff55.1%24 Mar 2025aider.chat
35gemini-2.5-flash-preview-05-20_23KGoogle DeepMindclosedEdit format: diff55.1%25 May 2025aider.chat
36o3-mini-2025-01-31_mediumOpenAIclosedeffort medium; Edit format: diff53.8%31 Jan 2025aider.chat
37grok-3-betaxAIclosedEdit format: diff53.3%10 Apr 2025aider.chat
38gpt-4.1-2025-04-14OpenAIclosedEdit format: diff52.4%14 Apr 2025aider.chat
39claude-3-5-sonnet-20241022AnthropicclosedEdit format: diff51.6%17 Jan 2025aider.chat
40grok-3-mini-beta_highxAIclosedeffort high; Edit format: whole49.3%10 Apr 2025aider.chat
41deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeekopenEdit format: diff48.4%not datedAider LLM Leaderboards
42gemini-2.5-flash-preview-04-17Google DeepMindclosedEdit format: diff47.1%20 Apr 2025aider.chat
43chatgpt-4o-03-27-2025OpenAIclosedEdit format: diff45.3%29 Mar 2025aider.chat
44gpt-4.5-preview-2025-02-27OpenAIclosedEdit format: diff44.9%27 Feb 2025aider.chat
45gemini-2.5-flash-preview-05-20Google DeepMindclosedEdit format: diff44.0%26 May 2025aider.chat
46openai/gpt-oss-120bEpoch: gpt-oss-120b_high, OpenAIopeneffort high; Edit format: diff41.8%not datedAider LLM Leaderboards, via Epoch AI
47openai/gpt-oss-120bEpoch: openai/gpt-oss-120b_high, OpenAIopeneffort high; Edit format: diff41.8%6 Aug 2025aider.chat
48Qwen/Qwen3-32BEpoch: Qwen3-32B, AlibabaopenEdit format: diff40.0%8 May 2025aider.chat
49gemini-exp-1206Google DeepMind,GoogleclosedEdit format: whole38.2%22 Dec 2024aider.chat
50gemini-2.0-pro-exp-02-05Google DeepMindclosedEdit format: whole35.6%25 Feb 2025aider.chat
51grok-3-mini-beta_lowxAIclosedeffort low; Edit format: whole34.7%10 Apr 2025aider.chat
52o1-mini-2024-09-12_unknownOpenAIclosedeffort unknown; Edit format: whole32.9%22 Dec 2024aider.chat
53gpt-4.1-mini-2025-04-14OpenAIclosedEdit format: diff32.4%14 Apr 2025aider.chat
54claude-3-5-haiku-20241022AnthropicclosedEdit format: diff28.0%21 Dec 2024aider.chat
55chatgpt-4o-01-29-2025OpenAIclosedEdit format: diff27.1%15 Feb 2025aider.chat
56gpt-4o-2024-08-06OpenAIclosedEdit format: diff23.1%30 Dec 2024aider.chat
57gemini-2.0-flash-expGoogle DeepMind,GoogleclosedEdit format: whole22.2%22 Dec 2024aider.chat
58qwen-max-2025-01-25AlibabaclosedEdit format: diff21.8%28 Jan 2025aider.chat
59Qwen/QwQ-32BEpoch: QwQ-32B, AlibabaopenEdit format: diff20.9%6 Mar 2025aider.chat
60gemini-2.0-flash-thinking-exp-01-21Google DeepMind,GoogleclosedEdit format: diff18.2%21 Jan 2025aider.chat
61gpt-4o-2024-11-20OpenAIclosedEdit format: diff18.2%30 Dec 2024aider.chat
62deepseek-ai/DeepSeek-V2.5Epoch: DeepSeek-V2.5, DeepSeekopenEdit format: diff17.8%21 Dec 2024aider.chat
63Qwen/Qwen2.5-Coder-32B-InstructEpoch: Qwen2.5-Coder-32B-Instruct, AlibabaopenEdit format: whole16.4%not datedAider LLM Leaderboards
64meta-llama/Llama-4-Maverick-17B-128E-InstructEpoch: Llama-4-Maverick-17B-128E-Instruct, Meta AIopenEdit format: whole15.6%6 Apr 2025aider.chat
65yi-lightning01.AIclosedEdit format: whole12.9%23 Dec 2024aider.chat
66CohereLabs/c4ai-command-a-03-2025Epoch: c4ai-command-a-03-2025, CohereopenEdit format: whole12.0%14 Mar 2025aider.chat
67codestral-2501Mistral AIno page matchopenEdit format: whole11.1%13 Jan 2025aider.chat
68openhands-lm-32b-v0.1not statedEdit format: whole10.2%19 Apr 2025aider.chat
69gpt-4.1-nano-2025-04-14OpenAIclosedEdit format: whole8.9%14 Apr 2025aider.chat
70Qwen/Qwen2.5-Coder-32B-InstructEpoch: Qwen2.5-Coder-32B-Instruct, AlibabaopenEdit format: diff8.0%not datedAider LLM Leaderboards
71google/gemma-3-27b-itEpoch: gemma-3-27b-it, Google DeepMindopenEdit format: whole4.9%15 Mar 2025aider.chat
72gpt-4o-mini-2024-07-18OpenAIclosedEdit format: whole3.6%21 Dec 2024aider.chat

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method