The Agent Company
ReportedNo description yet: we only describe an eval set when we can do it correctly.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Score shown
- Epoch's column "% Resolved", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
- Results
- 16 in all: 6 for open-weight models, 10 for closed ones. 6 linked to a model page here.
Best open-weight result: deepseek-ai/DeepSeek-V3.2-Exp, 42.9%, number 1 of 16 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | deepseek-ai/DeepSeek-V3.2-ExpEpoch: DeepSeek-V3.2-Exp, DeepSeek | open | Environment model: Qwen_Plus | 42.9% | 13 Oct 2025 | TheAgentCompany leaderboard |
| 2 | gemini-2.5-flash-preview-09-2025Google DeepMind | closed | Environment model: GPT-4o | 41.1% | 13 Oct 2025 | TheAgentCompany leaderboard |
| 3 | claude-sonnet-4-20250514Anthropic | closed | Environment model: Claude 3.5 Sonnet | 33.1% | 14 Jun 2025 | TheAgentCompany leaderboard |
| 4 | claude-3-7-sonnet-20250219Anthropic | closed | Environment model: Claude 3.5 Sonnet | 30.9% | 14 Jun 2025 | TheAgentCompany leaderboard |
| 5 | gemini-2.5-pro-preview-05-06Google DeepMind | closed | Environment model: Claude 3.5 Sonnet | 30.3% | 10 May 2025 | TheAgentCompany leaderboard |
| 6 | claude-3-7-sonnet-20250219Anthropic | closed | Environment model: Claude 3.5 Sonnet | 26.3% | 10 May 2025 | TheAgentCompany leaderboard |
| 7 | claude-3-5-sonnet-20241022Anthropic | closed | Environment model: Claude 3.5 Sonnet | 24.0% | 17 Dec 2024 | TheAgentCompany experiment results github |
| 8 | gemini-2.0-flash-001Google DeepMind,Google | closed | Environment model: Claude 3.5 Sonnet | 11.4% | 17 Dec 2024 | TheAgentCompany experiment results github |
| 9 | gpt-4o-2024-11-20OpenAI | closed | Environment model: Claude 3.5 Sonnet | 8.6% | 17 Dec 2024 | TheAgentCompany experiment results github |
| 10 | meta-llama/Llama-3.1-405B-InstructEpoch: Llama-3.1-405B-Instruct, Meta AI | open | Environment model: Claude 3.5 Sonnet | 7.4% | 17 Dec 2024 | TheAgentCompany experiment results github |
| 11 | meta-llama/Llama-3.1-70B-InstructEpoch: Llama-3.1-70B-Instruct, Meta AI | open | Environment model: Claude 3.5 Sonnet | 6.9% | 17 Dec 2024 | TheAgentCompany experiment results github |
| 12 | Qwen/Qwen2.5-72B-InstructEpoch: qwen2.5-72b-instruct, Alibaba | open | Environment model: Claude 3.5 Sonnet | 5.7% | 17 Dec 2024 | TheAgentCompany experiment results github |
| 13 | gemini-1.5-pro-002Google DeepMind | closed | Environment model: Claude 3.5 Sonnet | 3.4% | 17 Dec 2024 | TheAgentCompany experiment results github |
| 14 | amazon.nova-pro-v1:0Amazon | closed | Environment model: Claude 3.5 Sonnet | 1.7% | 17 Dec 2024 | TheAgentCompany experiment results github |
| 15 | meta-llama/Llama-3.1-70B-InstructEpoch: Llama-3.1-70B-Instruct, Meta AI | open | Environment model: Claude 3.5 Sonnet | 1.7% | 17 Dec 2024 | TheAgentCompany experiment results github |
| 16 | Qwen/Qwen2-72B-InstructEpoch: qwen2-72b-instruct, Alibaba | open | Environment model: Claude 3.5 Sonnet | 1.1% | 17 Dec 2024 | TheAgentCompany experiment results github |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method