GeoBench
ReportedNo description yet: we only describe an eval set when we can do it correctly.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Link
- https://geobench.org/
- Score shown
- Epoch's column "ACW Country %", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
- Results
- 51 in all: 6 for open-weight models, 45 for closed ones. 5 linked to a model page here.
Best open-weight result: Qwen2.5-VL-72B-Instruct, 62.0%, number 37 of 51 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | claude-fable-5_highAnthropic | closed | effort high; Tools: None | 89.0% | not dated | geobench.org |
| 2 | gemini-3-flash-previewGoogle DeepMind | closed | Tools: None | 88.0% | not dated | geobench.org |
| 3 | gemini-2.5-pro-preview-05-06Google DeepMind | closed | Tools: None | 86.0% | not dated | geobench.org |
| 4 | gemini-3.8-flash_unknownGoogle DeepMind | closed | effort unknown; Tools: None | 86.0% | not dated | geobench.org |
| 5 | muse-spark-1.3_unknownMeta AI | closed | effort unknown; Tools: None | 86.0% | not dated | geobench.org |
| 6 | gemini-3.5-flash_unknownGoogle DeepMind | closed | effort unknown; Tools: None | 85.0% | not dated | geobench.org |
| 7 | claude-fable-5-1_highAnthropic | closed | effort high; Tools: None | 84.0% | not dated | geobench.org |
| 8 | gemini-2.5-pro-exp-03-25Google DeepMind | closed | Tools: Search | 84.0% | not dated | geobench.org |
| 9 | gemini-3-pro-previewGoogle DeepMind | closed | Tools: None | 84.0% | not dated | geobench.org |
| 10 | gemini-3.1-pro-previewGoogle DeepMind | closed | Tools: None | 84.0% | not dated | geobench.org |
| 11 | gemini-3.1-flash-liteGoogle | closed | Tools: None | 83.0% | not dated | geobench.org |
| 12 | claude-opus-4-8_highAnthropic | closed | effort high; Tools: None | 81.0% | not dated | geobench.org |
| 13 | gemini-2.5-pro-preview-05-06Google DeepMind | closed | Tools: Search | 81.0% | not dated | geobench.org |
| 14 | gemini-2.5-pro-preview-06-05Google DeepMind | closed | Tools: None | 81.0% | not dated | geobench.org |
| 15 | gpt-5-2025-08-07_mediumOpenAI | closed | effort medium; Tools: None | 81.0% | not dated | geobench.org |
| 16 | o1-2024-12-17_mediumOpenAI | closed | effort medium; Tools: None | 80.0% | not dated | GeoBench leaderboard |
| 17 | gemini-2.5-pro-exp-03-25Google DeepMind | closed | Tools: None | 78.0% | not dated | geobench.org |
| 18 | gemini-2.0-pro-exp-02-05Google DeepMind | closed | Tools: None | 77.5% | not dated | geobench.org |
| 19 | gemini-2.0-flash-001Google DeepMind,Google | closed | Tools: None | 77.0% | not dated | geobench.org |
| 20 | gemini-1.5-flash-002Google DeepMind | closed | Tools: None | 76.0% | not dated | GeoBench leaderboard |
| 21 | gemini-2.5-flash-preview-05-20Google DeepMind | closed | Tools: None | 76.0% | not dated | geobench.org |
| 22 | claude-opus-4-5-20251101_32KAnthropic | closed | Tools: None | 75.0% | not dated | geobench.org |
| 23 | claude-sonnet-5_highAnthropic | closed | effort high; Tools: None | 74.0% | not dated | geobench.org |
| 24 | o3-2025-04-16_mediumOpenAI | closed | effort medium; Tools: None | 74.0% | not dated | GeoBench leaderboard |
| 25 | claude-sonnet-4-6_32KAnthropic | closed | Tools: None | 73.0% | not dated | geobench.org |
| 26 | gemini-2.5-flash-preview-04-17Google DeepMind | closed | Tools: None | 73.0% | not dated | GeoBench leaderboard |
| 27 | gpt-5.6-sol_highOpenAI | closed | effort high; Tools: None | 73.0% | not dated | geobench.org |
| 28 | gpt-4.1-2025-04-14OpenAI | closed | Tools: None | 72.0% | not dated | GeoBench leaderboard |
| 29 | gpt-4o-2024-11-20OpenAI | closed | Tools: None | 71.0% | not dated | GeoBench leaderboard |
| 30 | gemini-2.0-flash-thinking-exp-01-21Google DeepMind,Google | closed | Tools: None | 70.0% | not dated | geobench.org |
| 31 | claude-3-7-sonnet-20250219Anthropic | closed | Tools: None | 68.0% | not dated | GeoBench leaderboard |
| 32 | claude-3-7-sonnet-20250219_15KAnthropic | closed | Tools: None | 65.0% | not dated | geobench.org |
| 33 | gpt-4o-mini-2024-07-18OpenAI | closed | Tools: None | 64.0% | not dated | GeoBench leaderboard |
| 34 | o4-mini-2025-04-16_highOpenAI | closed | effort high; Tools: None | 64.0% | not dated | GeoBench leaderboard |
| 35 | o4-mini-2025-04-16_mediumOpenAI | closed | effort medium; Tools: None | 64.0% | not dated | GeoBench leaderboard |
| 36 | claude-3-5-sonnet-20241022Anthropic | closed | Tools: None | 62.0% | not dated | GeoBench leaderboard |
| 37 | Qwen2.5-VL-72B-InstructAlibabano page match | open | Tools: None | 62.0% | not dated | GeoBench leaderboard |
| 38 | grok-4.7_unknownxAI | closed | effort unknown; Tools: None | 61.0% | not dated | geobench.org |
| 39 | claude-sonnet-4-5-20250929_32KAnthropic | closed | Tools: None | 60.0% | not dated | geobench.org |
| 40 | o3-2025-04-16_highOpenAI | closed | effort high; Tools: None | 60.0% | not dated | GeoBench leaderboard |
| 41 | grok-4.3_unknownxAI | closed | effort unknown; Tools: None | 56.0% | not dated | geobench.org |
| 42 | google/gemma-3-27b-itEpoch: gemma-3-27b-it, Google DeepMind | open | Tools: None | 52.0% | not dated | GeoBench leaderboard |
| 43 | meta-llama/Llama-3.2-90B-Vision-InstructEpoch: Llama-3.2-90B-Vision-Instruct, Meta AI | open | Tools: None | 52.0% | not dated | GeoBench leaderboard |
| 44 | meta-llama/Llama-4-Maverick-17B-128E-InstructEpoch: Llama-4-Maverick-17B-128E-Instruct, Meta AI | open | Tools: None | 52.0% | not dated | GeoBench leaderboard |
| 45 | claude-opus-4-20250514_32KAnthropic | closed | Tools: None | 49.0% | not dated | GeoBench leaderboard |
| 46 | grok-4-0709xAI | closed | Tools: None | 45.0% | not dated | geobench.org |
| 47 | Qwen/Qwen3.8-27BEpoch: qwen3.8-27b, Alibaba | open | Tools: None | 38.0% | not dated | geobench.org |
| 48 | claude-sonnet-4-20250514Anthropic | closed | Tools: None | 37.0% | not dated | GeoBench leaderboard |
| 49 | mistralai/Pixtral-12B-2409Epoch: Pixtral-12B-2409, Mistral AI | open | Tools: None | 37.0% | not dated | GeoBench leaderboard |
| 50 | claude-3-5-haiku-20241022Anthropic | closed | Tools: None | 34.0% | not dated | GeoBench leaderboard |
| 51 | claude-sonnet-4-20250514_32KAnthropic | closed | Tools: None | 32.0% | not dated | GeoBench leaderboard |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method