Aider polyglot
ReportedCoding exercises in several programming languages, where the model must edit code so that the exercise's tests pass.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Link
- https://aider.chat/docs/leaderboards/#polyglot-leaderboard
- Score shown
- Epoch's column "Percent correct", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
- Results
- 72 in all: 23 for open-weight models, 48 for closed ones, 1 where Epoch does not say. 19 linked to a model page here.
Best open-weight result: deepseek-reasoner, 74.2%, number 13 of 72 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | gpt-5-2025-08-07_highOpenAI | closed | effort high; Edit format: diff | 88.0% | 23 Aug 2025 | aider.chat |
| 2 | gpt-5-2025-08-07_mediumOpenAI | closed | effort medium; Edit format: diff | 86.7% | 25 Aug 2025 | aider.chat |
| 3 | o3-pro-2025-06-10_highOpenAI | closed | effort high; Edit format: diff | 84.9% | 28 Jun 2025 | aider.chat |
| 4 | gemini-2.5-pro-preview-06-05_32KGoogle DeepMind | closed | Edit format: diff-fenced | 83.1% | 6 Jun 2025 | aider.chat |
| 5 | gpt-5-2025-08-07_lowOpenAI | closed | effort low; Edit format: diff | 81.3% | 25 Aug 2025 | aider.chat |
| 6 | o3-2025-04-16_highOpenAI | closed | effort high; Edit format: diff | 81.3% | 25 Jun 2025 | aider.chat |
| 7 | grok-4-0709xAI | closed | Edit format: diff | 79.6% | not dated | Aider LLM Leaderboards |
| 8 | grok-4-0709_highxAI | closed | effort high; Edit format: diff | 79.6% | 11 Jul 2025 | aider.chat |
| 9 | gemini-2.5-pro-preview-06-05Google DeepMind | closed | Edit format: diff-fenced | 79.1% | 6 Jun 2025 | aider.chat |
| 10 | gemini-2.5-pro-preview-05-06Google DeepMind | closed | Edit format: diff-fenced | 76.9% | 7 May 2025 | aider.chat |
| 11 | o3-2025-04-16_mediumOpenAI | closed | effort medium; Edit format: diff | 76.9% | not dated | Aider LLM Leaderboards |
| 12 | o3-2025-04-16_unknownOpenAI | closed | effort unknown; Edit format: diff | 76.9% | 25 Jun 2025 | aider.chat |
| 13 | deepseek-reasonerDeepSeekno page match | open | Edit format: diff | 74.2% | 3 Oct 2025 | aider.chat |
| 14 | deepseek-ai/DeepSeek-V3.2-ExpEpoch: DeepSeek-V3.2-Exp_thinking, DeepSeek | open | effort thinking; Edit format: diff | 74.2% | not dated | Aider LLM Leaderboards |
| 15 | gemini-2.5-pro-exp-03-25Google DeepMind | closed | Edit format: diff-fenced | 72.9% | not dated | Aider LLM Leaderboards |
| 16 | gemini-2.5-pro-preview-03-25Google DeepMind | closed | Edit format: diff-fenced | 72.9% | 12 Apr 2025 | aider.chat |
| 17 | claude-opus-4-20250514_32KAnthropic | closed | Edit format: diff | 72.0% | 25 May 2025 | aider.chat |
| 18 | o4-mini-2025-04-16_highOpenAI | closed | effort high; Edit format: diff | 72.0% | 16 Apr 2025 | aider.chat |
| 19 | deepseek-ai/DeepSeek-R1-0528Epoch: DeepSeek-R1-0528, DeepSeek | open | Edit format: diff | 71.4% | 6 Jun 2025 | aider.chat |
| 20 | claude-opus-4-20250514Anthropic | closed | Edit format: diff | 70.7% | 25 May 2025 | aider.chat |
| 21 | deepseek-chatDeepSeekno page match | open | Edit format: diff | 70.2% | 3 Oct 2025 | aider.chat |
| 22 | deepseek-ai/DeepSeek-V3.2-ExpEpoch: DeepSeek-V3.2-Exp, DeepSeek | open | Edit format: diff | 70.2% | not dated | Aider LLM Leaderboards |
| 23 | claude-3-7-sonnet-20250219_32KAnthropic | closed | Edit format: diff | 64.9% | 24 Feb 2025 | aider.chat |
| 24 | o1-2024-12-17_highOpenAI | closed | effort high; Edit format: diff | 61.7% | 21 Dec 2024 | aider.chat |
| 25 | claude-sonnet-4-20250514_32KAnthropic | closed | Edit format: diff | 61.3% | 24 May 2025 | aider.chat |
| 26 | claude-3-7-sonnet-20250219Anthropic | closed | Edit format: diff | 60.4% | 24 Feb 2025 | aider.chat |
| 27 | o3-mini-2025-01-31_highOpenAI | closed | effort high; Edit format: diff | 60.4% | 31 Jan 2025 | aider.chat |
| 28 | Qwen/Qwen3-235B-A22BEpoch: Qwen3-235B-A22B, Alibaba | open | Edit format: diff | 59.6% | not dated | Aider LLM Leaderboards |
| 29 | Qwen/Qwen3-235B-A22B-Instruct-2507Epoch: Qwen3-235B-A22B-Instruct-2507, Alibaba | open | Edit format: diff | 59.6% | 9 May 2025 | aider.chat |
| 30 | moonshotai/Kimi-K2-InstructEpoch: Kimi-K2-Instruct, Moonshot | open | Edit format: diff | 59.1% | not dated | Aider LLM Leaderboards |
| 31 | moonshotai/kimi-k2-0905Moonshotno page match | open | Edit format: diff | 59.1% | 17 Jul 2025 | aider.chat |
| 32 | deepseek-ai/DeepSeek-R1Epoch: DeepSeek-R1, DeepSeek | open | Edit format: diff | 56.9% | 20 Jan 2025 | aider.chat |
| 33 | claude-sonnet-4-20250514Anthropic | closed | Edit format: diff | 56.4% | 24 May 2025 | aider.chat |
| 34 | deepseek-ai/DeepSeek-V3-0324Epoch: DeepSeek-V3-0324, DeepSeek | open | Edit format: diff | 55.1% | 24 Mar 2025 | aider.chat |
| 35 | gemini-2.5-flash-preview-05-20_23KGoogle DeepMind | closed | Edit format: diff | 55.1% | 25 May 2025 | aider.chat |
| 36 | o3-mini-2025-01-31_mediumOpenAI | closed | effort medium; Edit format: diff | 53.8% | 31 Jan 2025 | aider.chat |
| 37 | grok-3-betaxAI | closed | Edit format: diff | 53.3% | 10 Apr 2025 | aider.chat |
| 38 | gpt-4.1-2025-04-14OpenAI | closed | Edit format: diff | 52.4% | 14 Apr 2025 | aider.chat |
| 39 | claude-3-5-sonnet-20241022Anthropic | closed | Edit format: diff | 51.6% | 17 Jan 2025 | aider.chat |
| 40 | grok-3-mini-beta_highxAI | closed | effort high; Edit format: whole | 49.3% | 10 Apr 2025 | aider.chat |
| 41 | deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeek | open | Edit format: diff | 48.4% | not dated | Aider LLM Leaderboards |
| 42 | gemini-2.5-flash-preview-04-17Google DeepMind | closed | Edit format: diff | 47.1% | 20 Apr 2025 | aider.chat |
| 43 | chatgpt-4o-03-27-2025OpenAI | closed | Edit format: diff | 45.3% | 29 Mar 2025 | aider.chat |
| 44 | gpt-4.5-preview-2025-02-27OpenAI | closed | Edit format: diff | 44.9% | 27 Feb 2025 | aider.chat |
| 45 | gemini-2.5-flash-preview-05-20Google DeepMind | closed | Edit format: diff | 44.0% | 26 May 2025 | aider.chat |
| 46 | openai/gpt-oss-120bEpoch: gpt-oss-120b_high, OpenAI | open | effort high; Edit format: diff | 41.8% | not dated | Aider LLM Leaderboards, via Epoch AI |
| 47 | openai/gpt-oss-120bEpoch: openai/gpt-oss-120b_high, OpenAI | open | effort high; Edit format: diff | 41.8% | 6 Aug 2025 | aider.chat |
| 48 | Qwen/Qwen3-32BEpoch: Qwen3-32B, Alibaba | open | Edit format: diff | 40.0% | 8 May 2025 | aider.chat |
| 49 | gemini-exp-1206Google DeepMind,Google | closed | Edit format: whole | 38.2% | 22 Dec 2024 | aider.chat |
| 50 | gemini-2.0-pro-exp-02-05Google DeepMind | closed | Edit format: whole | 35.6% | 25 Feb 2025 | aider.chat |
| 51 | grok-3-mini-beta_lowxAI | closed | effort low; Edit format: whole | 34.7% | 10 Apr 2025 | aider.chat |
| 52 | o1-mini-2024-09-12_unknownOpenAI | closed | effort unknown; Edit format: whole | 32.9% | 22 Dec 2024 | aider.chat |
| 53 | gpt-4.1-mini-2025-04-14OpenAI | closed | Edit format: diff | 32.4% | 14 Apr 2025 | aider.chat |
| 54 | claude-3-5-haiku-20241022Anthropic | closed | Edit format: diff | 28.0% | 21 Dec 2024 | aider.chat |
| 55 | chatgpt-4o-01-29-2025OpenAI | closed | Edit format: diff | 27.1% | 15 Feb 2025 | aider.chat |
| 56 | gpt-4o-2024-08-06OpenAI | closed | Edit format: diff | 23.1% | 30 Dec 2024 | aider.chat |
| 57 | gemini-2.0-flash-expGoogle DeepMind,Google | closed | Edit format: whole | 22.2% | 22 Dec 2024 | aider.chat |
| 58 | qwen-max-2025-01-25Alibaba | closed | Edit format: diff | 21.8% | 28 Jan 2025 | aider.chat |
| 59 | Qwen/QwQ-32BEpoch: QwQ-32B, Alibaba | open | Edit format: diff | 20.9% | 6 Mar 2025 | aider.chat |
| 60 | gemini-2.0-flash-thinking-exp-01-21Google DeepMind,Google | closed | Edit format: diff | 18.2% | 21 Jan 2025 | aider.chat |
| 61 | gpt-4o-2024-11-20OpenAI | closed | Edit format: diff | 18.2% | 30 Dec 2024 | aider.chat |
| 62 | deepseek-ai/DeepSeek-V2.5Epoch: DeepSeek-V2.5, DeepSeek | open | Edit format: diff | 17.8% | 21 Dec 2024 | aider.chat |
| 63 | Qwen/Qwen2.5-Coder-32B-InstructEpoch: Qwen2.5-Coder-32B-Instruct, Alibaba | open | Edit format: whole | 16.4% | not dated | Aider LLM Leaderboards |
| 64 | meta-llama/Llama-4-Maverick-17B-128E-InstructEpoch: Llama-4-Maverick-17B-128E-Instruct, Meta AI | open | Edit format: whole | 15.6% | 6 Apr 2025 | aider.chat |
| 65 | yi-lightning01.AI | closed | Edit format: whole | 12.9% | 23 Dec 2024 | aider.chat |
| 66 | CohereLabs/c4ai-command-a-03-2025Epoch: c4ai-command-a-03-2025, Cohere | open | Edit format: whole | 12.0% | 14 Mar 2025 | aider.chat |
| 67 | codestral-2501Mistral AIno page match | open | Edit format: whole | 11.1% | 13 Jan 2025 | aider.chat |
| 68 | openhands-lm-32b-v0.1 | not stated | Edit format: whole | 10.2% | 19 Apr 2025 | aider.chat |
| 69 | gpt-4.1-nano-2025-04-14OpenAI | closed | Edit format: whole | 8.9% | 14 Apr 2025 | aider.chat |
| 70 | Qwen/Qwen2.5-Coder-32B-InstructEpoch: Qwen2.5-Coder-32B-Instruct, Alibaba | open | Edit format: diff | 8.0% | not dated | Aider LLM Leaderboards |
| 71 | google/gemma-3-27b-itEpoch: gemma-3-27b-it, Google DeepMind | open | Edit format: whole | 4.9% | 15 Mar 2025 | aider.chat |
| 72 | gpt-4o-mini-2024-07-18OpenAI | closed | Edit format: whole | 3.6% | 21 Dec 2024 | aider.chat |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method