Skip to content
Weights Not Walls
Menu

The Agent Company

Reported

No description yet: we only describe an eval set when we can do it correctly.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Score shown
Epoch's column "% Resolved", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
Results
16 in all: 6 for open-weight models, 10 for closed ones. 6 linked to a model page here.

Best open-weight result: deepseek-ai/DeepSeek-V3.2-Exp, 42.9%, number 1 of 16 results.

The Agent Company results
#ModelWeightsSettingScoreDateSource
1deepseek-ai/DeepSeek-V3.2-ExpEpoch: DeepSeek-V3.2-Exp, DeepSeekopenEnvironment model: Qwen_Plus42.9%13 Oct 2025TheAgentCompany leaderboard
2gemini-2.5-flash-preview-09-2025Google DeepMindclosedEnvironment model: GPT-4o41.1%13 Oct 2025TheAgentCompany leaderboard
3claude-sonnet-4-20250514AnthropicclosedEnvironment model: Claude 3.5 Sonnet33.1%14 Jun 2025TheAgentCompany leaderboard
4claude-3-7-sonnet-20250219AnthropicclosedEnvironment model: Claude 3.5 Sonnet30.9%14 Jun 2025TheAgentCompany leaderboard
5gemini-2.5-pro-preview-05-06Google DeepMindclosedEnvironment model: Claude 3.5 Sonnet30.3%10 May 2025TheAgentCompany leaderboard
6claude-3-7-sonnet-20250219AnthropicclosedEnvironment model: Claude 3.5 Sonnet26.3%10 May 2025TheAgentCompany leaderboard
7claude-3-5-sonnet-20241022AnthropicclosedEnvironment model: Claude 3.5 Sonnet24.0%17 Dec 2024TheAgentCompany experiment results github
8gemini-2.0-flash-001Google DeepMind,GoogleclosedEnvironment model: Claude 3.5 Sonnet11.4%17 Dec 2024TheAgentCompany experiment results github
9gpt-4o-2024-11-20OpenAIclosedEnvironment model: Claude 3.5 Sonnet8.6%17 Dec 2024TheAgentCompany experiment results github
10meta-llama/Llama-3.1-405B-InstructEpoch: Llama-3.1-405B-Instruct, Meta AIopenEnvironment model: Claude 3.5 Sonnet7.4%17 Dec 2024TheAgentCompany experiment results github
11meta-llama/Llama-3.1-70B-InstructEpoch: Llama-3.1-70B-Instruct, Meta AIopenEnvironment model: Claude 3.5 Sonnet6.9%17 Dec 2024TheAgentCompany experiment results github
12Qwen/Qwen2.5-72B-InstructEpoch: qwen2.5-72b-instruct, AlibabaopenEnvironment model: Claude 3.5 Sonnet5.7%17 Dec 2024TheAgentCompany experiment results github
13gemini-1.5-pro-002Google DeepMindclosedEnvironment model: Claude 3.5 Sonnet3.4%17 Dec 2024TheAgentCompany experiment results github
14amazon.nova-pro-v1:0AmazonclosedEnvironment model: Claude 3.5 Sonnet1.7%17 Dec 2024TheAgentCompany experiment results github
15meta-llama/Llama-3.1-70B-InstructEpoch: Llama-3.1-70B-Instruct, Meta AIopenEnvironment model: Claude 3.5 Sonnet1.7%17 Dec 2024TheAgentCompany experiment results github
16Qwen/Qwen2-72B-InstructEpoch: qwen2-72b-instruct, AlibabaopenEnvironment model: Claude 3.5 Sonnet1.1%17 Dec 2024TheAgentCompany experiment results github

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method