Skip to content
Weights Not Walls
Menu

Balrog

Reported

No description yet: we only describe an eval set when we can do it correctly.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Link
https://balrogai.com/
Score shown
Epoch's column "Average progress", as a percentage, higher is better; random guessing scores about 0.0%. Error bars as the source gives them.
Results
41 in all: 16 for open-weight models, 25 for closed ones. 16 linked to a model page here.

Best open-weight result: deepseek-ai/DeepSeek-R1, 34.9%, number 15 of 41 results.

Balrog results
#ModelWeightsSettingScoreDateSource
1gpt-6-astra_maxOpenAIclosedeffort max68.3%±2.018 Sep 2026balrogai.com
2claude-opus-5_maxAnthropicclosedeffort max63.4%±1.820 Sep 2026balrogai.com
3gpt-5.6-sol_maxOpenAIclosedeffort max60.0%±2.018 Sep 2026balrogai.com
4gemini-3-pro-previewGoogle DeepMindcloseddefault58.1%±2.13 Feb 2026balrogai.com
5gemini-3.1-pro-previewGoogle DeepMindcloseddefault57.0%±2.025 Feb 2026Balrog Leaderboard
6gemini-3.1-pro-previewGoogle DeepMindcloseddefault56.9%±2.021 Feb 2026Balrog Leaderboard
7gpt-5.6-terra_maxOpenAIclosedeffort max53.2%±2.020 Sep 2026balrogai.com
8gemini-3-flash-previewGoogle DeepMindcloseddefault48.1%±2.413 Feb 2026balrogai.com
9gpt-5.6-luna_maxOpenAIclosedeffort max45.6%±1.718 Sep 2026balrogai.com
10grok-4-0709xAIcloseddefault43.6%±2.223 Jul 2025balrogai.com
11claude-opus-4-5-20251101Anthropiccloseddefault43.5%±2.324 Feb 2026Balrog Leaderboard
12gemini-2.5-pro-exp-03-25Google DeepMindcloseddefault43.3%±2.325 Apr 2025balrogai.com
13claude-opus-4-5-20251101_64KAnthropiccloseddefault43.0%±2.324 Feb 2026Balrog Leaderboard
14claude-opus-4-5-20251101_unknownAnthropicclosedeffort unknown43.0%±2.324 Feb 2026balrogai.com
15deepseek-ai/DeepSeek-R1Epoch: DeepSeek-R1, DeepSeekopendefault34.9%±2.110 Apr 2025balrogai.com
16gemini-2.5-flashGoogle DeepMindcloseddefault33.5%±2.222 Jul 2025balrogai.com
17gpt-5-2025-08-07_minimalOpenAIclosedeffort minimal32.8%±2.219 Aug 2025balrogai.com
18claude-3-5-sonnet-20241022Anthropiccloseddefault32.6%±1.911 Nov 2024balrogai.com
19gpt-4o-2024-05-13OpenAIcloseddefault32.3%±1.511 Nov 2024balrogai.com
20claude-haiku-4-5-20251001Anthropiccloseddefault31.2%±2.124 Feb 2026Balrog Leaderboard
21claude-haiku-4-5-20251001_1KAnthropiccloseddefault31.2%±2.124 Feb 2026balrogai.com
22grok-3-betaxAIcloseddefault29.5%±2.225 Apr 2025balrogai.com
23RekaAI/reka-flash-3Epoch: reka-flash-3, Reka AIopendefault29.2%±1.813 Mar 2025balrogai.com
24meta-llama/Llama-3.1-70B-InstructEpoch: Llama-3.1-70B-Instruct, Meta AIopendefault27.9%±1.411 Nov 2024balrogai.com
25meta-llama/Llama-3.2-90B-Vision-InstructEpoch: Llama-3.2-90B-Vision-Instruct, Meta AIopendefault27.3%±1.411 Nov 2024balrogai.com
26meta-llama/Llama-3.3-70B-InstructEpoch: Llama-3.3-70B-Instruct, Meta AIopendefault23.0%±1.79 Dec 2024balrogai.com
27gemini-1.5-pro-002Google DeepMindcloseddefault21.0%±1.211 Nov 2024balrogai.com
28deepseek-ai/DeepSeek-R1-Distill-Qwen-32BEpoch: DeepSeek-R1-Distill-Qwen-32B, DeepSeekopendefault19.5%±1.626 Jan 2025balrogai.com
29claude-3-5-haiku-20241022Anthropiccloseddefault19.3%±1.811 Dec 2024balrogai.com
30mistralai/Mistral-Nemo-Instruct-2407Epoch: Mistral-Nemo-Instruct-2407, Mistral AIopendefault17.6%±1.59 Dec 2024balrogai.com
31gpt-4o-mini-2024-07-18OpenAIcloseddefault17.4%±1.411 Nov 2024balrogai.com
32meta-llama/Llama-3.2-11B-Vision-InstructEpoch: Llama-3.2-11B-Vision-Instruct, Meta AIopendefault16.8%±1.511 Nov 2024balrogai.com
33Qwen/Qwen2.5-72B-InstructEpoch: qwen2.5-72b-instruct, Alibabaopendefault16.2%±1.625 Nov 2024balrogai.com
34meta-llama/Llama-3.1-8B-InstructEpoch: Llama-3.1-8B-Instruct, Meta AIopendefault15.1%±1.611 Nov 2024balrogai.com
35gemini-1.5-flash-002Google DeepMindcloseddefault14.6%±1.411 Nov 2024balrogai.com
36Qwen/Qwen2-VL-72B-InstructEpoch: Qwen2-VL-72B-Instructopendefault12.8%±1.625 Nov 2024balrogai.com
37microsoft/phi-4Epoch: phi-4, Microsoft Researchopendefault11.6%±1.413 Jan 2025balrogai.com
38meta-llama/Llama-3.2-3B-InstructEpoch: Llama-3.2-3B-Instruct, Meta AIopendefault10.1%±1.311 Nov 2024balrogai.com
39Qwen/Qwen2.5-7B-InstructEpoch: qwen2.5-7b-instruct, Alibabaopendefault7.8%±1.125 Nov 2024balrogai.com
40meta-llama/Llama-3.2-1B-InstructEpoch: Llama-3.2-1B-Instruct, Meta AIopendefault6.6%±1.011 Nov 2024balrogai.com
41Qwen/Qwen2-VL-7B-InstructEpoch: Qwen2-VL-7B-Instructopendefault3.7%±0.825 Nov 2024balrogai.com

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method