BBH
ReportedBIG-Bench Hard: a set of the hardest tasks from the BIG-Bench collection, covering logic, arithmetic and language puzzles.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Score shown
- Epoch's column "Average", as a percentage, higher is better; random guessing scores about 25.0%. The source gives no error bars.
- Results
- 88 in all: 78 for open-weight models, 8 for closed ones, 2 where Epoch does not say. 33 linked to a model page here.
Best open-weight result: deepseek-ai/DeepSeek-V3, 87.5%, number 2 of 88 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | gemini-1.5-pro-001Google DeepMind | closed | Shots: 3 | 89.2% | not dated | Gemini 1.5 report |
| 2 | deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeek | open | Shots: 3 | 87.5% | not dated | DeepSeek-V3 Technical Report |
| 3 | gemini-1.5-pro-001-feb24Google DeepMind | closed | Shots: 3 | 84.0% | not dated | Gemini 1.5 report |
| 4 | deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeek | open | Shots: 3 | 82.9% | not dated | DeepSeek-V3 Technical Report |
| 5 | Llama-3.1-405BMeta AIno page match | open | Shots: 3 | 82.9% | not dated | DeepSeek-V3 Technical Report |
| 6 | Llama-3.1-405BMeta AIno page match | open | Shots: 3 | 82.9% | not dated | DeepSeek-V3 Technical Report |
| 7 | microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, Microsoft | open | Shots: 3 | 81.4% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 8 | Qwen2.5-72BAlibabano page match | open | Shots: 3 | 79.8% | not dated | DeepSeek-V3 Technical Report |
| 9 | Qwen2.5-72BAlibabano page match | open | Shots: 3 | 79.8% | not dated | DeepSeek-V3 Technical Report |
| 10 | microsoft/Phi-3-small-8k-instructEpoch: Phi-3-small-8k-instruct, Microsoft | open | Shots: 3 | 79.1% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your PhonePhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 11 | DeepSeek-V2DeepSeekno page match | open | Shots: 3 | 78.8% | not dated | DeepSeek-V3 Technical Report |
| 12 | DeepSeek-V2DeepSeekno page match | open | Shots: 3 | 78.8% | not dated | DeepSeek-V3 Technical Report |
| 13 | gpt-4-0613OpenAI | closed | Shots: 3 | 75.1% | not dated | Baichuan 2: Open Large-scale Language Models |
| 14 | microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, Microsoft | open | Shots: 3 | 71.7% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 15 | 01-ai/Yi-34B-ChatEpoch: Yi-34B-Chat, 01.AI | open | Shots: 3 | 71.7% | not dated | Yi: Open Foundation Models by 01.AI |
| 16 | stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AI | open | Shots: 3 | 69.3% | not dated | Qwen Technical Report |
| 17 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 3 | 64.9% | not dated | Qwen Technical Report |
| 18 | gpt-3.5-turbo-0613OpenAI | closed | Shots: 3 | 61.6% | not dated | Baichuan 2: Open Large-scale Language Models |
| 19 | microsoft/phi-2Epoch: phi-2, Microsoft | open | Shots: 3 | 59.4% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 20 | Nemotron-4 15BNvidia | closed | Shots: 3 | 58.7% | not dated | Nemotron-4 15B Technical Report |
| 21 | meta-llama/Llama-2-70b-chatEpoch: Llama-2-70b-chat, Meta AI | open | Shots: 3 | 58.5% | not dated | Yi: Open Foundation Models by 01.AI |
| 22 | LLaMA-65BMeta AIno page match | open | Shots: 3 | 58.4% | not dated | Qwen Technical Report |
| 23 | meta-llama/Llama-2-13b-chatEpoch: Llama-2-13b-chat, Meta AI | open | Shots: 3 | 58.2% | not dated | Yi: Open Foundation Models by 01.AI |
| 24 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | default | 56.1% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 25 | gemma-7bGoogle DeepMindno page match | open | default | 55.1% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 26 | gemma-7bGoogle DeepMindno page match | open | Shots: 3 | 55.1% | not dated | Nemotron-4 15B Technical Report |
| 27 | Qwen/Qwen-14B-ChatEpoch: Qwen-14B-Chat, Alibaba | open | Shots: 3 | 55.0% | not dated | Yi: Open Foundation Models by 01.AI |
| 28 | Yi-34B01.AIno page match | open | Shots: 3 | 54.3% | not dated | Yi: Open Foundation Models by 01.AI |
| 29 | Qwen-14BAlibabano page match | open | Shots: 3 | 53.4% | not dated | Qwen Technical Report |
| 30 | Qwen-14BAlibabano page match | open | Shots: 3 | 53.4% | not dated | Nemotron-4 15B Technical Report |
| 31 | Qwen-14BAlibabano page match | open | Shots: 3 | 53.4% | not dated | Qwen Technical Report |
| 32 | internlm/internlm-20bEpoch: internlm-20b | open | Shots: 3 | 52.5% | not dated | Qwen Technical Report |
| 33 | 01-ai/Yi-34B-ChatEpoch: Yi-34B-Chat, 01.AI | open | Shots: 0 | 51.4% | not dated | Yi: Open Foundation Models by 01.AI |
| 34 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 3 | 51.2% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 35 | LLaMA-33BMeta AIno page match | open | Shots: 3 | 50.0% | not dated | Qwen Technical Report |
| 36 | Qwen/Qwen-14B-ChatEpoch: Qwen-14B-Chat, Alibaba | open | Shots: 0 | 49.7% | not dated | Yi: Open Foundation Models by 01.AI |
| 37 | Baichuan-2-13B-BaseBaichuanno page match | open | Shots: 3 | 49.0% | not dated | Qwen Technical Report |
| 38 | Baichuan-2-13B-BaseBaichuanno page match | open | Shots: 3 | 48.8% | not dated | Nemotron-4 15B Technical Report |
| 39 | Baichuan-2-13B-BaseBaichuanno page match | open | Shots: 3 | 48.8% | not dated | Baichuan 2: Open Large-scale Language Models |
| 40 | baichuan-inc/Baichuan2-13B-ChatEpoch: Baichuan2-13B-Chat | open | Shots: 3 | 47.2% | not dated | Yi: Open Foundation Models by 01.AI |
| 41 | 01-ai/Yi-6B-ChatEpoch: Yi-6B-Chat, 01.AI | open | Shots: 3 | 47.2% | not dated | Yi: Open Foundation Models by 01.AI |
| 42 | Llama-2-13bMeta AIno page match | open | Shots: 3 | 47.0% | not dated | Baichuan 2: Open Large-scale Language Models |
| 43 | Qwen-7BAlibabano page match | open | Shots: 3 | 45.0% | not dated | Qwen Technical Report |
| 44 | Qwen-7BAlibabano page match | open | Shots: 3 | 45.0% | not dated | Qwen Technical Report |
| 45 | Llama-2-34bMeta AI | closed | Shots: 3 | 44.1% | not dated | Nemotron-4 15B Technical Report |
| 46 | Llama-2-34bMeta AI | closed | Shots: 3 | 44.1% | not dated | Qwen Technical Report |
| 47 | Llama-2-34bMeta AI | closed | Shots: 3 | 44.1% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 48 | LLaMA-65BMeta AIno page match | open | Shots: 3 | 43.5% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 49 | vicuna-13b-v1.1 | not stated | Shots: 3 | 43.0% | not dated | Baichuan 2: Open Large-scale Language Models |
| 50 | baichuan-inc/Baichuan-13B-BaseEpoch: Baichuan-13B-Base, Baichuan | open | Shots: 3 | 43.0% | not dated | Baichuan 2: Open Large-scale Language Models |
| 51 | Yi-6B01.AIno page match | open | Shots: 3 | 42.8% | not dated | Yi: Open Foundation Models by 01.AI |
| 52 | internlm/internlm-chat-20bEpoch: internlm-chat-20b | open | Shots: 0 | 42.4% | not dated | Yi: Open Foundation Models by 01.AI |
| 53 | meta-llama/Llama-2-70b-chatEpoch: Llama-2-70b-chat, Meta AI | open | Shots: 0 | 42.4% | not dated | Yi: Open Foundation Models by 01.AI |
| 54 | Baichuan-2-7B-BaseBaichuanno page match | open | Shots: 3 | 41.6% | not dated | Qwen Technical Report |
| 55 | Baichuan-2-7B-BaseBaichuanno page match | open | Shots: 3 | 41.6% | not dated | Baichuan 2: Open Large-scale Language Models |
| 56 | LLaMA-33BMeta AIno page match | open | Shots: 3 | 39.8% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 57 | 01-ai/Yi-6B-ChatEpoch: Yi-6B-Chat, 01.AI | open | Shots: 0 | 39.7% | not dated | Yi: Open Foundation Models by 01.AI |
| 58 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 3 | 39.5% | not dated | Nemotron-4 15B Technical Report |
| 59 | Llama-2-13bMeta AIno page match | open | default | 39.4% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 60 | Llama-2-13bMeta AIno page match | open | Shots: 3 | 39.4% | not dated | Nemotron-4 15B Technical Report |
| 61 | Llama-2-13bMeta AIno page match | open | Shots: 3 | 39.4% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 62 | Llama-2-7bMeta AIno page match | open | Shots: 3 | 39.2% | not dated | Baichuan 2: Open Large-scale Language Models |
| 63 | baichuan-inc/Baichuan2-13B-ChatEpoch: Baichuan2-13B-Chat | open | Shots: 0 | 38.8% | not dated | Yi: Open Foundation Models by 01.AI |
| 64 | mpt-30bMosaicMLno page match | open | Shots: 3 | 38.0% | not dated | Qwen Technical Report |
| 65 | LLaMA-13BMeta AIno page match | open | Shots: 3 | 37.9% | not dated | Qwen Technical Report |
| 66 | LLaMA-13BMeta AIno page match | open | Shots: 3 | 37.9% | not dated | Baichuan 2: Open Large-scale Language Models |
| 67 | falcon-40bTechnology Innovation Instituteno page match | open | Shots: 3 | 37.1% | not dated | Qwen Technical Report |
| 68 | internlm/internlm-7bEpoch: internlm-7b | open | Shots: 3 | 37.0% | not dated | Qwen Technical Report |
| 69 | LLaMA-13BMeta AIno page match | open | Shots: 3 | 37.0% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 70 | internlm/internlm-chat-20bEpoch: internlm-chat-20b | open | Shots: 3 | 36.7% | not dated | Yi: Open Foundation Models by 01.AI |
| 71 | mpt-7bMosaicMLno page match | open | Shots: 3 | 35.6% | not dated | Qwen Technical Report |
| 72 | gemma-2bGoogle DeepMindno page match | open | default | 35.2% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 73 | mpt-7bMosaicMLno page match | open | Shots: 3 | 35.2% | not dated | Baichuan 2: Open Large-scale Language Models |
| 74 | INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AI | not stated | default | 34.8% | not dated | INTELLECT-1 Technical Report |
| 75 | zai-org/chatglm2-6bEpoch: chatglm2-6b | open | Shots: 3 | 33.7% | not dated | Qwen Technical Report |
| 76 | zai-org/chatglm2-6bEpoch: chatglm2-6b | open | Shots: 3 | 33.7% | not dated | Baichuan 2: Open Large-scale Language Models |
| 77 | LLaMA-7BMeta AIno page match | open | Shots: 3 | 33.5% | not dated | Qwen Technical Report |
| 78 | meta-llama/Llama-2-13b-chatEpoch: Llama-2-13b-chat, Meta AI | open | Shots: 0 | 32.9% | not dated | Yi: Open Foundation Models by 01.AI |
| 79 | Llama-2-7bMeta AIno page match | open | default | 32.6% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 80 | Llama-2-7bMeta AIno page match | open | Shots: 3 | 32.6% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 81 | baichuan-inc/Baichuan-7BEpoch: Baichuan-7B, Baichuan | open | Shots: 3 | 32.5% | not dated | Baichuan 2: Open Large-scale Language Models |
| 82 | LLaMA-7BMeta AIno page match | open | Shots: 3 | 32.4% | not dated | Baichuan 2: Open Large-scale Language Models |
| 83 | mpt-7bMosaicMLno page match | open | Shots: 3 | 31.0% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 84 | LLaMA-7BMeta AIno page match | open | Shots: 3 | 30.3% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 85 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 3 | 28.8% | not dated | Baichuan 2: Open Large-scale Language Models |
| 86 | Qwen/Qwen-1_8BEpoch: Qwen-1_8B | open | Shots: 3 | 28.2% | not dated | Qwen Technical Report |
| 87 | Qwen/Qwen-1_8BEpoch: Qwen-1_8B | open | Shots: 3 | 28.2% | not dated | Qwen Technical Report |
| 88 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 3 | 28.0% | not dated | Qwen Technical Report |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method