Balrog
ReportedNo description yet: we only describe an eval set when we can do it correctly.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Link
- https://balrogai.com/
- Score shown
- Epoch's column "Average progress", as a percentage, higher is better; random guessing scores about 0.0%. Error bars as the source gives them.
- Results
- 41 in all: 16 for open-weight models, 25 for closed ones. 16 linked to a model page here.
Best open-weight result: deepseek-ai/DeepSeek-R1, 34.9%, number 15 of 41 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | gpt-6-astra_maxOpenAI | closed | effort max | 68.3%±2.0 | 18 Sep 2026 | balrogai.com |
| 2 | claude-opus-5_maxAnthropic | closed | effort max | 63.4%±1.8 | 20 Sep 2026 | balrogai.com |
| 3 | gpt-5.6-sol_maxOpenAI | closed | effort max | 60.0%±2.0 | 18 Sep 2026 | balrogai.com |
| 4 | gemini-3-pro-previewGoogle DeepMind | closed | default | 58.1%±2.1 | 3 Feb 2026 | balrogai.com |
| 5 | gemini-3.1-pro-previewGoogle DeepMind | closed | default | 57.0%±2.0 | 25 Feb 2026 | Balrog Leaderboard |
| 6 | gemini-3.1-pro-previewGoogle DeepMind | closed | default | 56.9%±2.0 | 21 Feb 2026 | Balrog Leaderboard |
| 7 | gpt-5.6-terra_maxOpenAI | closed | effort max | 53.2%±2.0 | 20 Sep 2026 | balrogai.com |
| 8 | gemini-3-flash-previewGoogle DeepMind | closed | default | 48.1%±2.4 | 13 Feb 2026 | balrogai.com |
| 9 | gpt-5.6-luna_maxOpenAI | closed | effort max | 45.6%±1.7 | 18 Sep 2026 | balrogai.com |
| 10 | grok-4-0709xAI | closed | default | 43.6%±2.2 | 23 Jul 2025 | balrogai.com |
| 11 | claude-opus-4-5-20251101Anthropic | closed | default | 43.5%±2.3 | 24 Feb 2026 | Balrog Leaderboard |
| 12 | gemini-2.5-pro-exp-03-25Google DeepMind | closed | default | 43.3%±2.3 | 25 Apr 2025 | balrogai.com |
| 13 | claude-opus-4-5-20251101_64KAnthropic | closed | default | 43.0%±2.3 | 24 Feb 2026 | Balrog Leaderboard |
| 14 | claude-opus-4-5-20251101_unknownAnthropic | closed | effort unknown | 43.0%±2.3 | 24 Feb 2026 | balrogai.com |
| 15 | deepseek-ai/DeepSeek-R1Epoch: DeepSeek-R1, DeepSeek | open | default | 34.9%±2.1 | 10 Apr 2025 | balrogai.com |
| 16 | gemini-2.5-flashGoogle DeepMind | closed | default | 33.5%±2.2 | 22 Jul 2025 | balrogai.com |
| 17 | gpt-5-2025-08-07_minimalOpenAI | closed | effort minimal | 32.8%±2.2 | 19 Aug 2025 | balrogai.com |
| 18 | claude-3-5-sonnet-20241022Anthropic | closed | default | 32.6%±1.9 | 11 Nov 2024 | balrogai.com |
| 19 | gpt-4o-2024-05-13OpenAI | closed | default | 32.3%±1.5 | 11 Nov 2024 | balrogai.com |
| 20 | claude-haiku-4-5-20251001Anthropic | closed | default | 31.2%±2.1 | 24 Feb 2026 | Balrog Leaderboard |
| 21 | claude-haiku-4-5-20251001_1KAnthropic | closed | default | 31.2%±2.1 | 24 Feb 2026 | balrogai.com |
| 22 | grok-3-betaxAI | closed | default | 29.5%±2.2 | 25 Apr 2025 | balrogai.com |
| 23 | RekaAI/reka-flash-3Epoch: reka-flash-3, Reka AI | open | default | 29.2%±1.8 | 13 Mar 2025 | balrogai.com |
| 24 | meta-llama/Llama-3.1-70B-InstructEpoch: Llama-3.1-70B-Instruct, Meta AI | open | default | 27.9%±1.4 | 11 Nov 2024 | balrogai.com |
| 25 | meta-llama/Llama-3.2-90B-Vision-InstructEpoch: Llama-3.2-90B-Vision-Instruct, Meta AI | open | default | 27.3%±1.4 | 11 Nov 2024 | balrogai.com |
| 26 | meta-llama/Llama-3.3-70B-InstructEpoch: Llama-3.3-70B-Instruct, Meta AI | open | default | 23.0%±1.7 | 9 Dec 2024 | balrogai.com |
| 27 | gemini-1.5-pro-002Google DeepMind | closed | default | 21.0%±1.2 | 11 Nov 2024 | balrogai.com |
| 28 | deepseek-ai/DeepSeek-R1-Distill-Qwen-32BEpoch: DeepSeek-R1-Distill-Qwen-32B, DeepSeek | open | default | 19.5%±1.6 | 26 Jan 2025 | balrogai.com |
| 29 | claude-3-5-haiku-20241022Anthropic | closed | default | 19.3%±1.8 | 11 Dec 2024 | balrogai.com |
| 30 | mistralai/Mistral-Nemo-Instruct-2407Epoch: Mistral-Nemo-Instruct-2407, Mistral AI | open | default | 17.6%±1.5 | 9 Dec 2024 | balrogai.com |
| 31 | gpt-4o-mini-2024-07-18OpenAI | closed | default | 17.4%±1.4 | 11 Nov 2024 | balrogai.com |
| 32 | meta-llama/Llama-3.2-11B-Vision-InstructEpoch: Llama-3.2-11B-Vision-Instruct, Meta AI | open | default | 16.8%±1.5 | 11 Nov 2024 | balrogai.com |
| 33 | Qwen/Qwen2.5-72B-InstructEpoch: qwen2.5-72b-instruct, Alibaba | open | default | 16.2%±1.6 | 25 Nov 2024 | balrogai.com |
| 34 | meta-llama/Llama-3.1-8B-InstructEpoch: Llama-3.1-8B-Instruct, Meta AI | open | default | 15.1%±1.6 | 11 Nov 2024 | balrogai.com |
| 35 | gemini-1.5-flash-002Google DeepMind | closed | default | 14.6%±1.4 | 11 Nov 2024 | balrogai.com |
| 36 | Qwen/Qwen2-VL-72B-InstructEpoch: Qwen2-VL-72B-Instruct | open | default | 12.8%±1.6 | 25 Nov 2024 | balrogai.com |
| 37 | microsoft/phi-4Epoch: phi-4, Microsoft Research | open | default | 11.6%±1.4 | 13 Jan 2025 | balrogai.com |
| 38 | meta-llama/Llama-3.2-3B-InstructEpoch: Llama-3.2-3B-Instruct, Meta AI | open | default | 10.1%±1.3 | 11 Nov 2024 | balrogai.com |
| 39 | Qwen/Qwen2.5-7B-InstructEpoch: qwen2.5-7b-instruct, Alibaba | open | default | 7.8%±1.1 | 25 Nov 2024 | balrogai.com |
| 40 | meta-llama/Llama-3.2-1B-InstructEpoch: Llama-3.2-1B-Instruct, Meta AI | open | default | 6.6%±1.0 | 11 Nov 2024 | balrogai.com |
| 41 | Qwen/Qwen2-VL-7B-InstructEpoch: Qwen2-VL-7B-Instruct | open | default | 3.7%±0.8 | 25 Nov 2024 | balrogai.com |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method