Skip to content
Weights Not Walls
Menu

BBH

Reported

BIG-Bench Hard: a set of the hardest tasks from the BIG-Bench collection, covering logic, arithmetic and language puzzles.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Score shown
Epoch's column "Average", as a percentage, higher is better; random guessing scores about 25.0%. The source gives no error bars.
Results
88 in all: 78 for open-weight models, 8 for closed ones, 2 where Epoch does not say. 33 linked to a model page here.

Best open-weight result: deepseek-ai/DeepSeek-V3, 87.5%, number 2 of 88 results.

BBH results
#ModelWeightsSettingScoreDateSource
1gemini-1.5-pro-001Google DeepMindclosedShots: 389.2%not datedGemini 1.5 report
2deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeekopenShots: 387.5%not datedDeepSeek-V3 Technical Report
3gemini-1.5-pro-001-feb24Google DeepMindclosedShots: 384.0%not datedGemini 1.5 report
4deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeekopenShots: 382.9%not datedDeepSeek-V3 Technical Report
5Llama-3.1-405BMeta AIno page matchopenShots: 382.9%not datedDeepSeek-V3 Technical Report
6Llama-3.1-405BMeta AIno page matchopenShots: 382.9%not datedDeepSeek-V3 Technical Report
7microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, MicrosoftopenShots: 381.4%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
8Qwen2.5-72BAlibabano page matchopenShots: 379.8%not datedDeepSeek-V3 Technical Report
9Qwen2.5-72BAlibabano page matchopenShots: 379.8%not datedDeepSeek-V3 Technical Report
10microsoft/Phi-3-small-8k-instructEpoch: Phi-3-small-8k-instruct, MicrosoftopenShots: 379.1%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your PhonePhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
11DeepSeek-V2DeepSeekno page matchopenShots: 378.8%not datedDeepSeek-V3 Technical Report
12DeepSeek-V2DeepSeekno page matchopenShots: 378.8%not datedDeepSeek-V3 Technical Report
13gpt-4-0613OpenAIclosedShots: 375.1%not datedBaichuan 2: Open Large-scale Language Models
14microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, MicrosoftopenShots: 371.7%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
1501-ai/Yi-34B-ChatEpoch: Yi-34B-Chat, 01.AIopenShots: 371.7%not datedYi: Open Foundation Models by 01.AI
16stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AIopenShots: 369.3%not datedQwen Technical Report
17meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 364.9%not datedQwen Technical Report
18gpt-3.5-turbo-0613OpenAIclosedShots: 361.6%not datedBaichuan 2: Open Large-scale Language Models
19microsoft/phi-2Epoch: phi-2, MicrosoftopenShots: 359.4%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
20Nemotron-4 15BNvidiaclosedShots: 358.7%not datedNemotron-4 15B Technical Report
21meta-llama/Llama-2-70b-chatEpoch: Llama-2-70b-chat, Meta AIopenShots: 358.5%not datedYi: Open Foundation Models by 01.AI
22LLaMA-65BMeta AIno page matchopenShots: 358.4%not datedQwen Technical Report
23meta-llama/Llama-2-13b-chatEpoch: Llama-2-13b-chat, Meta AIopenShots: 358.2%not datedYi: Open Foundation Models by 01.AI
24mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopendefault56.1%not datedGemma: Open Models Based on Gemini Research and Technology
25gemma-7bGoogle DeepMindno page matchopendefault55.1%not datedGemma: Open Models Based on Gemini Research and Technology
26gemma-7bGoogle DeepMindno page matchopenShots: 355.1%not datedNemotron-4 15B Technical Report
27Qwen/Qwen-14B-ChatEpoch: Qwen-14B-Chat, AlibabaopenShots: 355.0%not datedYi: Open Foundation Models by 01.AI
28Yi-34B01.AIno page matchopenShots: 354.3%not datedYi: Open Foundation Models by 01.AI
29Qwen-14BAlibabano page matchopenShots: 353.4%not datedQwen Technical Report
30Qwen-14BAlibabano page matchopenShots: 353.4%not datedNemotron-4 15B Technical Report
31Qwen-14BAlibabano page matchopenShots: 353.4%not datedQwen Technical Report
32internlm/internlm-20bEpoch: internlm-20bopenShots: 352.5%not datedQwen Technical Report
3301-ai/Yi-34B-ChatEpoch: Yi-34B-Chat, 01.AIopenShots: 051.4%not datedYi: Open Foundation Models by 01.AI
34meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 351.2%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
35LLaMA-33BMeta AIno page matchopenShots: 350.0%not datedQwen Technical Report
36Qwen/Qwen-14B-ChatEpoch: Qwen-14B-Chat, AlibabaopenShots: 049.7%not datedYi: Open Foundation Models by 01.AI
37Baichuan-2-13B-BaseBaichuanno page matchopenShots: 349.0%not datedQwen Technical Report
38Baichuan-2-13B-BaseBaichuanno page matchopenShots: 348.8%not datedNemotron-4 15B Technical Report
39Baichuan-2-13B-BaseBaichuanno page matchopenShots: 348.8%not datedBaichuan 2: Open Large-scale Language Models
40baichuan-inc/Baichuan2-13B-ChatEpoch: Baichuan2-13B-ChatopenShots: 347.2%not datedYi: Open Foundation Models by 01.AI
4101-ai/Yi-6B-ChatEpoch: Yi-6B-Chat, 01.AIopenShots: 347.2%not datedYi: Open Foundation Models by 01.AI
42Llama-2-13bMeta AIno page matchopenShots: 347.0%not datedBaichuan 2: Open Large-scale Language Models
43Qwen-7BAlibabano page matchopenShots: 345.0%not datedQwen Technical Report
44Qwen-7BAlibabano page matchopenShots: 345.0%not datedQwen Technical Report
45Llama-2-34bMeta AIclosedShots: 344.1%not datedNemotron-4 15B Technical Report
46Llama-2-34bMeta AIclosedShots: 344.1%not datedQwen Technical Report
47Llama-2-34bMeta AIclosedShots: 344.1%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
48LLaMA-65BMeta AIno page matchopenShots: 343.5%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
49vicuna-13b-v1.1not statedShots: 343.0%not datedBaichuan 2: Open Large-scale Language Models
50baichuan-inc/Baichuan-13B-BaseEpoch: Baichuan-13B-Base, BaichuanopenShots: 343.0%not datedBaichuan 2: Open Large-scale Language Models
51Yi-6B01.AIno page matchopenShots: 342.8%not datedYi: Open Foundation Models by 01.AI
52internlm/internlm-chat-20bEpoch: internlm-chat-20bopenShots: 042.4%not datedYi: Open Foundation Models by 01.AI
53meta-llama/Llama-2-70b-chatEpoch: Llama-2-70b-chat, Meta AIopenShots: 042.4%not datedYi: Open Foundation Models by 01.AI
54Baichuan-2-7B-BaseBaichuanno page matchopenShots: 341.6%not datedQwen Technical Report
55Baichuan-2-7B-BaseBaichuanno page matchopenShots: 341.6%not datedBaichuan 2: Open Large-scale Language Models
56LLaMA-33BMeta AIno page matchopenShots: 339.8%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
5701-ai/Yi-6B-ChatEpoch: Yi-6B-Chat, 01.AIopenShots: 039.7%not datedYi: Open Foundation Models by 01.AI
58mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 339.5%not datedNemotron-4 15B Technical Report
59Llama-2-13bMeta AIno page matchopendefault39.4%not datedGemma: Open Models Based on Gemini Research and Technology
60Llama-2-13bMeta AIno page matchopenShots: 339.4%not datedNemotron-4 15B Technical Report
61Llama-2-13bMeta AIno page matchopenShots: 339.4%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
62Llama-2-7bMeta AIno page matchopenShots: 339.2%not datedBaichuan 2: Open Large-scale Language Models
63baichuan-inc/Baichuan2-13B-ChatEpoch: Baichuan2-13B-ChatopenShots: 038.8%not datedYi: Open Foundation Models by 01.AI
64mpt-30bMosaicMLno page matchopenShots: 338.0%not datedQwen Technical Report
65LLaMA-13BMeta AIno page matchopenShots: 337.9%not datedQwen Technical Report
66LLaMA-13BMeta AIno page matchopenShots: 337.9%not datedBaichuan 2: Open Large-scale Language Models
67falcon-40bTechnology Innovation Instituteno page matchopenShots: 337.1%not datedQwen Technical Report
68internlm/internlm-7bEpoch: internlm-7bopenShots: 337.0%not datedQwen Technical Report
69LLaMA-13BMeta AIno page matchopenShots: 337.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
70internlm/internlm-chat-20bEpoch: internlm-chat-20bopenShots: 336.7%not datedYi: Open Foundation Models by 01.AI
71mpt-7bMosaicMLno page matchopenShots: 335.6%not datedQwen Technical Report
72gemma-2bGoogle DeepMindno page matchopendefault35.2%not datedGemma: Open Models Based on Gemini Research and Technology
73mpt-7bMosaicMLno page matchopenShots: 335.2%not datedBaichuan 2: Open Large-scale Language Models
74INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AInot stateddefault34.8%not datedINTELLECT-1 Technical Report
75zai-org/chatglm2-6bEpoch: chatglm2-6bopenShots: 333.7%not datedQwen Technical Report
76zai-org/chatglm2-6bEpoch: chatglm2-6bopenShots: 333.7%not datedBaichuan 2: Open Large-scale Language Models
77LLaMA-7BMeta AIno page matchopenShots: 333.5%not datedQwen Technical Report
78meta-llama/Llama-2-13b-chatEpoch: Llama-2-13b-chat, Meta AIopenShots: 032.9%not datedYi: Open Foundation Models by 01.AI
79Llama-2-7bMeta AIno page matchopendefault32.6%not datedGemma: Open Models Based on Gemini Research and Technology
80Llama-2-7bMeta AIno page matchopenShots: 332.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
81baichuan-inc/Baichuan-7BEpoch: Baichuan-7B, BaichuanopenShots: 332.5%not datedBaichuan 2: Open Large-scale Language Models
82LLaMA-7BMeta AIno page matchopenShots: 332.4%not datedBaichuan 2: Open Large-scale Language Models
83mpt-7bMosaicMLno page matchopenShots: 331.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
84LLaMA-7BMeta AIno page matchopenShots: 330.3%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
85falcon-7bTechnology Innovation Instituteno page matchopenShots: 328.8%not datedBaichuan 2: Open Large-scale Language Models
86Qwen/Qwen-1_8BEpoch: Qwen-1_8BopenShots: 328.2%not datedQwen Technical Report
87Qwen/Qwen-1_8BEpoch: Qwen-1_8BopenShots: 328.2%not datedQwen Technical Report
88falcon-7bTechnology Innovation Instituteno page matchopenShots: 328.0%not datedQwen Technical Report

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method