MMLU
ReportedMultiple-choice questions across 57 subjects, from school level to professional level.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Score shown
- Epoch's column "EM", as a percentage, higher is better; random guessing scores about 25.0%. The source gives no error bars.
- Results
- 217 in all: 159 for open-weight models, 51 for closed ones, 7 where Epoch does not say. 79 linked to a model page here.
Best open-weight result: deepseek-ai/DeepSeek-V3, 87.2%, number 3 of 217 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | gpt-4o-2024-11-20OpenAI | closed | Shots: 5 | 88.1% | not dated | Phi-4 Technical Report |
| 2 | claude-3-5-sonnet-20241022Anthropic | closed | Shots: 5 | 87.3% | not dated | Stanford CRFM Leaderboard |
| 3 | deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeek | open | Shots: 5 | 87.2% | not dated | Stanford CRFM Leaderboard |
| 4 | deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeek | open | Shots: 5 | 87.1% | not dated | DeepSeek-V3 Technical Report |
| 5 | gemini-1.5-pro-002Google DeepMind | closed | Shots: 5 | 86.9% | not dated | Stanford CRFM Leaderboard |
| 6 | claude-3-5-sonnet-20240620Anthropic | closed | Shots: 5 | 86.5% | not dated | Stanford CRFM Leaderboard |
| 7 | gpt-4-0314OpenAI | closed | Shots: 5 | 86.4% | not dated | GPT-4 Technical Report |
| 8 | meta-llama/Llama-3.3-70B-InstructEpoch: Llama-3.3-70B-Instruct, Meta AI | open | Shots: 5 | 86.3% | not dated | Phi-4 Technical Report |
| 9 | gemini-1.5-pro-001Google DeepMind | closed | Shots: 5 | 85.9% | not dated | Gemini 1.5 Report |
| 10 | Qwen/Qwen2.5-72B-InstructEpoch: qwen2.5-72b-instruct, Alibaba | open | Shots: 5 | 85.3% | not dated | Phi-4 Technical Report |
| 11 | Qwen2.5-72BAlibabano page match | open | Shots: 5 | 85.0% | not dated | DeepSeek-V3 Technical Report |
| 12 | microsoft/phi-4Epoch: phi-4, Microsoft Research | open | Shots: 5 | 84.8% | not dated | Phi-4 Technical Report |
| 13 | claude-3-opus-20240229Anthropic | closed | Shots: 5 | 84.6% | not dated | Stanford CRFM Leaderboard |
| 14 | meta-llama/Llama-3.1-405B-InstructEpoch: Llama-3.1-405B-Instruct, Meta AI | open | Shots: 5 | 84.5% | not dated | Stanford CRFM Leaderboard |
| 15 | Llama-3.1-405BMeta AIno page match | open | Shots: 5 | 84.4% | not dated | DeepSeek-V3 Technical Report |
| 16 | gpt-4o-2024-08-06OpenAI | closed | Shots: 5 | 84.3% | not dated | Stanford CRFM Leaderboard |
| 17 | gpt-4o-2024-05-13OpenAI | closed | Shots: 5 | 84.2% | not dated | Stanford CRFM Leaderboard |
| 18 | Qwen/Qwen2.5-72B-InstructEpoch: qwen2.5-72b-instruct, Alibaba | open | Shots: 5 | 83.4% | not dated | Stanford CRFM Leaderboard |
| 19 | gemini-1.5-pro-001-feb24Google DeepMind | closed | Shots: 5 | 82.7% | not dated | Stanford CRFM Leaderboard |
| 20 | gpt-4-0613OpenAI | closed | Shots: 5 | 82.4% | not dated | Stanford CRFM Leaderboard |
| 21 | Qwen/Qwen2-72B-InstructEpoch: qwen2-72b-instruct, Alibaba | open | Shots: 5 | 82.4% | not dated | Stanford CRFM Leaderboard |
| 22 | amazon.nova-pro-v1:0Amazon | closed | Shots: 5 | 82.0% | not dated | Stanford CRFM Leaderboard |
| 23 | gemini-1.5-pro-001-feb24Google DeepMind | closed | Shots: 5 | 81.9% | not dated | Gemini 1.5 Report |
| 24 | gpt-4o-mini-2024-07-18OpenAI | closed | Shots: 5 | 81.8% | not dated | Phi-4 Technical Report |
| 25 | gpt-4-turbo-2024-04-09OpenAI | closed | Shots: 5 | 81.3% | not dated | Stanford CRFM Leaderboard |
| 26 | meta-llama/Llama-3.2-90B-Vision-InstructEpoch: Llama-3.2-90B-Vision-Instruct, Meta AI | open | Shots: 5 | 80.3% | not dated | Stanford CRFM Leaderboard |
| 27 | meta-llama/Llama-3.1-70B-InstructEpoch: Llama-3.1-70B-Instruct, Meta AI | open | Shots: 5 | 80.1% | not dated | Stanford CRFM Leaderboard |
| 28 | mistral-large-2407Mistral AIno page match | open | Shots: 5 | 80.0% | not dated | Stanford CRFM Leaderboard |
| 29 | Qwen/Qwen2.5-14B-InstructEpoch: qwen2.5-14b-instruct, Alibaba | open | Shots: 5 | 79.9% | not dated | Phi-4 Technical Report |
| 30 | gemini-2.0-flash-expGoogle DeepMind,Google | closed | Shots: 5 | 79.7% | not dated | Stanford CRFM Leaderboard |
| 31 | gpt-4-turboOpenAI | closed | Shots: 5 | 79.6% | not dated | Stanford CRFM Leaderboard |
| 32 | meta-llama/Meta-Llama-3-70B-InstructEpoch: Meta-Llama-3-70B-Instruct, Meta AI | open | Shots: 5 | 79.3% | not dated | Stanford CRFM Leaderboard |
| 33 | Yi-large01.AI | closed | Shots: 5 | 79.3% | not dated | Stanford CRFM Leaderboard |
| 34 | meta-llama/Llama-3.3-70B-InstructEpoch: Llama-3.3-70B-Instruct, Meta AI | open | Shots: 5 | 79.1% | not dated | Stanford CRFM Leaderboard |
| 35 | Qwen2.5-Coder-32BAlibabano page match | open | default | 79.1% | not dated | Qwen2.5-Coder Technical Report |
| 36 | claude-2.0Anthropic | closed | Shots: 5 | 78.5% | not dated | Model Card and Evaluations for Claude Models |
| 37 | DeepSeek-V2DeepSeekno page match | open | Shots: 5 | 78.4% | not dated | DeepSeek-V3 Technical Report |
| 38 | microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, Microsoft | open | Shots: 5 | 78.0% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 39 | gemini-1.5-flash-001Google DeepMind | closed | Shots: 5 | 77.9% | not dated | Stanford CRFM Leaderboard |
| 40 | microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, Microsoft | open | Shots: 5 | 77.9% | not dated | Phi-4 Technical Report |
| 41 | gemini-1.5-flash-0514Google DeepMind | closed | Shots: 5 | 77.8% | not dated | Stanford CRFM Leaderboard |
| 42 | mistralai/Mixtral-8x22B-v0.1Epoch: Mixtral-8x22B-v0.1, Mistral AI | open | Shots: 5 | 77.8% | not dated | Stanford CRFM Leaderboard |
| 43 | microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, Microsoft | open | Shots: 5 | 77.5% | not dated | Stanford CRFM Leaderboard |
| 44 | amazon.nova-lite-v1:0Amazon | closed | Shots: 5 | 77.0% | not dated | Stanford CRFM Leaderboard |
| 45 | claude-1.3Anthropic | closed | Shots: 5 | 77.0% | not dated | Model Card and Evaluations for Claude Models |
| 46 | gpt-4o-mini-2024-07-18OpenAI | closed | Shots: 5 | 76.7% | not dated | Stanford CRFM Leaderboard |
| 47 | Yi-34B01.AIno page match | open | Shots: 5 | 76.3% | not dated | Yi: Open Foundation Models by 01.AI |
| 48 | Yi-34B01.AIno page match | open | Shots: 5 | 76.2% | not dated | Stanford CRFM Leaderboard |
| 49 | claude-3-sonnet-20240229Anthropic | closed | Shots: 5 | 75.9% | not dated | Stanford CRFM Leaderboard |
| 50 | google/gemma-2-27b-itEpoch: gemma-2-27b-it, Google DeepMind | open | Shots: 5 | 75.7% | not dated | Stanford CRFM Leaderboard |
| 51 | microsoft/Phi-3-small-8k-instructEpoch: Phi-3-small-8k-instruct, Microsoft | open | Shots: 5 | 75.7% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 52 | Qwen/Qwen2.5-Coder-14BEpoch: Qwen2.5-Coder-14B | open | default | 75.2% | not dated | Qwen2.5-Coder Technical Report |
| 53 | qwen1.5-32BAlibabano page match | open | Shots: 5 | 74.4% | not dated | Stanford CRFM Leaderboard |
| 54 | claude-3-5-haiku-20241022Anthropic | closed | Shots: 5 | 74.3% | not dated | Stanford CRFM Leaderboard |
| 55 | gemini-1.5-flash-002Google DeepMind | closed | Shots: 5 | 73.9% | not dated | Stanford CRFM Leaderboard |
| 56 | claude-3-haiku-20240307Anthropic | closed | Shots: 5 | 73.8% | not dated | Stanford CRFM Leaderboard |
| 57 | claude-2.1Anthropic | closed | Shots: 5 | 73.5% | not dated | Stanford CRFM Leaderboard |
| 58 | 01-ai/Yi-34B-ChatEpoch: Yi-34B-Chat, 01.AI | open | Shots: 5 | 73.5% | not dated | Yi: Open Foundation Models by 01.AI |
| 59 | claude-instant-1.1Anthropic | closed | Shots: 5 | 73.4% | not dated | Model Card and Evaluations for Claude Models |
| 60 | claude-instant-1.1Anthropic | closed | Shots: 5 | 73.4% | not dated | Releasing Claude Instant 1.2 |
| 61 | claude-instant-1.2Anthropic | closed | Shots: 5 | 73.2% | not dated | Releasing Claude Instant 1.2 |
| 62 | Qwen/Qwen2.5-7B-InstructEpoch: qwen2.5-7b-instruct, Alibaba | open | Shots: 5 | 72.9% | not dated | Stanford CRFM Leaderboard |
| 63 | Inflection-1Inflection AI | closed | Shots: 5 | 72.7% | not dated | Inflection-1 |
| 64 | google/gemma-2-9b-itEpoch: gemma-2-9b-it, Google DeepMind | open | Shots: 5 | 72.1% | not dated | Stanford CRFM Leaderboard |
| 65 | gpt-3.5-turbo-1106OpenAI | closed | Shots: 5 | 71.4% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 66 | amazon.nova-micro-v1:0Amazon | closed | Shots: 5 | 70.8% | not dated | Stanford CRFM Leaderboard |
| 67 | falcon-180BTechnology Innovation Instituteno page match | open | Shots: 5 | 70.6% | not dated | The Falcon Series of Open Language Models |
| 68 | mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AI | open | Shots: 5 | 70.6% | not dated | Mixtral of Experts |
| 69 | mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AI | open | Shots: 5 | 70.5% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 70 | gemini-1.0-pro-001Google DeepMind | closed | Shots: 5 | 70.0% | not dated | Stanford CRFM Leaderboard |
| 71 | text-davinci-002OpenAI | closed | Shots: 5 | 70.0% | not dated | GPT-4 Technical Report |
| 72 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 5 | 69.9% | not dated | Mixtral of Experts |
| 73 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 5 | 69.8% | not dated | Qwen Technical Report |
| 74 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 5 | 69.5% | not dated | Stanford CRFM Leaderboard |
| 75 | CohereLabs/c4ai-command-r-plus-08-2024Epoch: c4ai-command-r-plus-08-2024, Cohere,Cohere Labs (formerly Cohere for AI) | open | Shots: 5 | 69.4% | not dated | Stanford CRFM Leaderboard |
| 76 | PaLM 540BGoogle Research | closed | Shots: 5 | 69.3% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 77 | gpt-3.5-turbo-0613OpenAI | closed | Shots: 5 | 68.9% | not dated | Stanford CRFM Leaderboard |
| 78 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 5 | 68.9% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 79 | claude-instant-1.2Anthropic | closed | Shots: 5 | 68.8% | not dated | Stanford CRFM Leaderboard |
| 80 | meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AI | open | Shots: 5 | 68.8% | not dated | Stanford CRFM Leaderboard |
| 81 | mistral-large-2402Mistral AI | closed | Shots: 5 | 68.8% | not dated | Stanford CRFM Leaderboard |
| 82 | microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, Microsoft | open | Shots: 5 | 68.8% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 83 | mistral-small-2402 | not stated | Shots: 5 | 68.7% | not dated | Stanford CRFM Leaderboard |
| 84 | qwen1.5-14BAlibabano page match | open | Shots: 5 | 68.6% | not dated | Stanford CRFM Leaderboard |
| 85 | stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AI | open | Shots: 5 | 68.6% | not dated | Qwen Technical Report |
| 86 | 01-ai/Yi-9BEpoch: Yi-9B | open | Shots: 5 | 68.4% | not dated | Yi: Open Foundation Models by 01.AI |
| 87 | Qwen2.5-Coder-7BAlibabano page match | open | default | 68.0% | not dated | Qwen2.5-Coder Technical Report |
| 88 | 01-ai/Yi-34B-ChatEpoch: Yi-34B-Chat, 01.AI | open | Shots: 0 | 67.6% | not dated | Yi: Open Foundation Models by 01.AI |
| 89 | Chinchilla (70B)DeepMind | closed | Shots: 5 | 67.5% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 90 | gpt-3.5-turbo-0125OpenAI | closed | Shots: 5 | 67.3% | not dated | Stanford CRFM Leaderboard |
| 91 | meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AI | open | Shots: 5 | 66.5% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 92 | Qwen-14BAlibabano page match | open | Shots: 5 | 66.3% | not dated | Nemotron-4 15B Technical Report |
| 93 | gemma-7bGoogle DeepMindno page match | open | Shots: 5 | 66.1% | not dated | Stanford CRFM Leaderboard |
| 94 | CohereLabs/c4ai-command-r-08-2024Epoch: c4ai-command-r-08-2024 | open | Shots: 5 | 65.2% | not dated | Stanford CRFM Leaderboard |
| 95 | Qwen/Qwen-14B-ChatEpoch: Qwen-14B-Chat, Alibaba | open | Shots: 5 | 65.0% | not dated | Yi: Open Foundation Models by 01.AI |
| 96 | gemma-7bGoogle DeepMindno page match | open | Shots: 5 | 64.3% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 97 | gemma-7bGoogle DeepMindno page match | open | Shots: 5 | 64.3% | not dated | Nemotron-4 15B Technical Report |
| 98 | bigcode/starcoder2-15bEpoch: starcoder2-15b, Hugging Face,ServiceNow,Nvidia,BigCode | open | default | 64.1% | not dated | Qwen2.5-Coder Technical Report |
| 99 | Qwen/Qwen-14B-ChatEpoch: Qwen-14B-Chat, Alibaba | open | Shots: 0 | 64.0% | not dated | Yi: Open Foundation Models by 01.AI |
| 100 | Yi-6B01.AIno page match | open | Shots: 5 | 64.0% | not dated | Stanford CRFM Leaderboard |
| 101 | gemma-7bGoogle DeepMindno page match | open | Shots: 5 | 63.6% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 102 | LLaMA-65BMeta AIno page match | open | Shots: 5 | 63.4% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 103 | LLaMA-65BMeta AIno page match | open | Shots: 5 | 63.4% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 104 | Yi-6B01.AIno page match | open | Shots: 5 | 63.2% | not dated | Yi: Open Foundation Models by 01.AI |
| 105 | Llama-2-34bMeta AI | closed | Shots: 5 | 62.6% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 106 | Llama-2-34bMeta AI | closed | Shots: 5 | 62.6% | not dated | Qwen Technical Report |
| 107 | Llama-2-34bMeta AI | closed | Shots: 5 | 62.6% | not dated | Nemotron-4 15B Technical Report |
| 108 | Qwen1.5-7BAlibabano page match | open | Shots: 5 | 62.6% | not dated | Stanford CRFM Leaderboard |
| 109 | mistralai/Mistral-7B-Instruct-v0.2Epoch: Mistral-7B-Instruct-v0.2, Mistral AI | open | Shots: 5 | 62.5% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 110 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 5 | 62.5% | not dated | Mixtral of Experts |
| 111 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 5 | 61.7% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 112 | Qwen2.5-Coder-32BAlibabano page match | open | default | 61.2% | not dated | Qwen2.5-Coder Technical Report |
| 113 | 01-ai/Yi-6B-ChatEpoch: Yi-6B-Chat, 01.AI | open | Shots: 5 | 61.0% | not dated | Yi: Open Foundation Models by 01.AI |
| 114 | deepseek-ai/DeepSeek-Coder-V2-Lite-BaseEpoch: DeepSeek-Coder-V2-Lite-Base | open | default | 60.5% | not dated | Qwen2.5-Coder Technical Report |
| 115 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 5 | 60.1% | not dated | Nemotron-4 15B Technical Report |
| 116 | Gopher (280B)DeepMind | closed | Shots: 5 | 60.0% | not dated | Scaling Language Models: Methods, Analysis & Insights from Training Gopher |
| 117 | Gopher (280B)DeepMind | closed | Shots: 5 | 60.0% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 118 | meta-llama/Llama-2-70b-chatEpoch: Llama-2-70b-chat, Meta AI | open | Shots: 5 | 59.9% | not dated | Yi: Open Foundation Models by 01.AI |
| 119 | mistralai/Mistral-7B-Instruct-v0.3Epoch: Mistral-7B-Instruct-v0.3, Mistral AI | open | Shots: 5 | 59.9% | not dated | Stanford CRFM Leaderboard |
| 120 | meta-llama/Llama-2-70b-chatEpoch: Llama-2-70b-chat, Meta AI | open | Shots: 0 | 59.4% | not dated | Yi: Open Foundation Models by 01.AI |
| 121 | Baichuan-2-13B-BaseBaichuanno page match | open | Shots: 5 | 59.2% | not dated | Nemotron-4 15B Technical Report |
| 122 | Baichuan-2-13B-BaseBaichuanno page match | open | Shots: 5 | 59.2% | not dated | Baichuan 2: Open Large-scale Language Models |
| 123 | LLaMA-33BMeta AIno page match | open | Shots: 5 | 58.7% | not dated | Qwen Technical Report |
| 124 | Nemotron-4 15BNvidia | closed | Shots: 5 | 58.7% | not dated | Nemotron-4 15B Technical Report |
| 125 | tiiuae/falcon-11BEpoch: falcon-11b, Technology Innovation Institute | open | Shots: 5 | 58.4% | not dated | Falcon2-11B Technical Report |
| 126 | microsoft/phi-2Epoch: phi-2, Microsoft | open | Shots: 5 | 58.4% | not dated | Stanford CRFM Leaderboard |
| 127 | 01-ai/Yi-6B-ChatEpoch: Yi-6B-Chat, 01.AI | open | Shots: 0 | 58.2% | not dated | Yi: Open Foundation Models by 01.AI |
| 128 | LLaMA-33BMeta AIno page match | open | Shots: 5 | 57.8% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 129 | LLaMA-33BMeta AIno page match | open | Shots: 5 | 57.8% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 130 | internlm/internlm-chat-20bEpoch: internlm-chat-20b | open | Shots: 5 | 57.4% | not dated | Yi: Open Foundation Models by 01.AI |
| 131 | falcon-40bTechnology Innovation Instituteno page match | open | Shots: 5 | 56.9% | not dated | Falcon2-11B Technical ReportFalcon2-11B Technical Report |
| 132 | LLaMA-33BMeta AIno page match | open | Shots: 5 | 56.8% | not dated | Mixtral of Experts |
| 133 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 5 | 56.6% | not dated | Stanford CRFM Leaderboard |
| 134 | meta-llama/Llama-3.2-11B-Vision-InstructEpoch: Llama-3.2-11B-Vision-Instruct, Meta AI | open | Shots: 5 | 56.5% | not dated | Stanford CRFM Leaderboard |
| 135 | microsoft/phi-2Epoch: phi-2, Microsoft | open | Shots: 5 | 56.3% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 136 | meta-llama/Llama-3.1-8B-InstructEpoch: Llama-3.1-8B-Instruct, Meta AI | open | Shots: 5 | 56.1% | not dated | Stanford CRFM Leaderboard |
| 137 | internlm/internlm-chat-20bEpoch: internlm-chat-20b | open | Shots: 0 | 55.6% | not dated | Yi: Open Foundation Models by 01.AI |
| 138 | Llama-2-13bMeta AIno page match | open | Shots: 5 | 55.6% | not dated | Mixtral of Experts |
| 139 | falcon-40bTechnology Innovation Instituteno page match | open | Shots: 5 | 55.4% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 140 | falcon-40bTechnology Innovation Instituteno page match | open | Shots: 5 | 55.4% | not dated | Qwen Technical Report |
| 141 | Llama-2-13bMeta AIno page match | open | Shots: 5 | 55.4% | not dated | Stanford CRFM Leaderboard |
| 142 | baichuan-inc/Baichuan2-13B-ChatEpoch: Baichuan2-13B-Chat | open | Shots: 0 | 55.1% | not dated | Yi: Open Foundation Models by 01.AI |
| 143 | Llama-2-13bMeta AIno page match | open | Shots: 5 | 55.1% | not dated | Baichuan 2: Open Large-scale Language Models |
| 144 | Llama-2-13bMeta AIno page match | open | Shots: 5 | 54.8% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 145 | Llama-2-13bMeta AIno page match | open | Shots: 5 | 54.8% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 146 | Llama-2-13bMeta AIno page match | open | Shots: 5 | 54.8% | not dated | Nemotron-4 15B Technical Report |
| 147 | Baichuan-2-7B-BaseBaichuanno page match | open | Shots: 5 | 54.2% | not dated | Baichuan 2: Open Large-scale Language Models |
| 148 | PaLM 62B | not stated | Shots: 5 | 53.7% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 149 | Qwen2.5-Coder-1.5BAlibabano page match | open | default | 53.6% | not dated | Qwen2.5-Coder Technical Report |
| 150 | Qwen-14BAlibabano page match | open | Shots: 5 | 53.4% | not dated | Qwen Technical Report |
| 151 | baichuan-inc/Baichuan-13B-BaseEpoch: Baichuan-13B-Base, Baichuan | open | Shots: 5 | 51.6% | not dated | Baichuan 2: Open Large-scale Language Models |
| 152 | internlm/internlm-7bEpoch: internlm-7b | open | Shots: 5 | 51.0% | not dated | Qwen Technical Report |
| 153 | meta-llama/Llama-2-13b-chatEpoch: Llama-2-13b-chat, Meta AI | open | Shots: 0 | 50.9% | not dated | Yi: Open Foundation Models by 01.AI |
| 154 | baichuan-inc/Baichuan2-13B-ChatEpoch: Baichuan2-13B-Chat | open | Shots: 5 | 50.1% | not dated | Yi: Open Foundation Models by 01.AI |
| 155 | INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AI | not stated | Shots: 5 | 49.9% | not dated | INTELLECT-1 Technical Report |
| 156 | zai-org/chatglm2-6bEpoch: chatglm2-6b | open | Shots: 5 | 47.9% | not dated | Qwen Technical Report |
| 157 | mpt-30bMosaicMLno page match | open | Shots: 5 | 47.9% | not dated | Qwen Technical Report |
| 158 | LLaMA-13BMeta AIno page match | open | Shots: 5 | 47.7% | not dated | Qwen Technical Report |
| 159 | meta-llama/Llama-2-13b-chatEpoch: Llama-2-13b-chat, Meta AI | open | Shots: 5 | 47.3% | not dated | Yi: Open Foundation Models by 01.AI |
| 160 | LLaMA-13BMeta AIno page match | open | Shots: 5 | 46.9% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 161 | LLaMA-13BMeta AIno page match | open | Shots: 5 | 46.9% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 162 | mpt-30bMosaicMLno page match | open | Shots: 5 | 46.9% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 163 | LLaMA-13BMeta AIno page match | open | Shots: 5 | 46.3% | not dated | Baichuan 2: Open Large-scale Language Models |
| 164 | Llama-2-7bMeta AIno page match | open | Shots: 5 | 45.8% | not dated | Stanford CRFM Leaderboard |
| 165 | Llama-2-7bMeta AIno page match | open | Shots: 5 | 45.7% | not dated | Baichuan 2: Open Large-scale Language Models |
| 166 | Llama-2-7bMeta AIno page match | open | Shots: 5 | 45.3% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 167 | Llama-2-7bMeta AIno page match | open | Shots: 5 | 45.3% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 168 | Llama-2-7bMeta AIno page match | open | Shots: 2 | 45.3% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 169 | Qwen-7BAlibabano page match | open | Shots: 5 | 45.0% | not dated | Qwen Technical Report |
| 170 | Llama-2-7bMeta AIno page match | open | Shots: 5 | 44.4% | not dated | Mixtral of Experts |
| 171 | text-davinci-001OpenAI | closed | Shots: 5 | 43.9% | not dated | Scaling Language Models: Methods, Analysis & Insights from Training Gopher |
| 172 | text-davinci-001OpenAI | closed | Shots: 5 | 43.9% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 173 | baichuan-inc/Baichuan-7BEpoch: Baichuan-7B, Baichuan | open | Shots: 5 | 42.3% | not dated | Baichuan 2: Open Large-scale Language Models |
| 174 | gemma-2bGoogle DeepMindno page match | open | Shots: 5 | 42.3% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 175 | Qwen/Qwen2.5-Coder-0.5BEpoch: Qwen2.5-Coder-0.5B | open | default | 42.0% | not dated | Qwen2.5-Coder Technical Report |
| 176 | Qwen/CodeQwen1.5-7BEpoch: CodeQwen1.5-7B | open | default | 40.5% | not dated | Qwen2.5-Coder Technical Report |
| 177 | deepseek-ai/deepseek-coder-33b-baseEpoch: deepseek-coder-33b-base, DeepSeek,Peking University | open | default | 39.4% | not dated | Qwen2.5-Coder Technical Report |
| 178 | bigcode/starcoder2-7bEpoch: starcoder2-7b, Hugging Face,ServiceNow,Nvidia,BigCode | open | default | 38.8% | not dated | Qwen2.5-Coder Technical Report |
| 179 | microsoft/phi-1_5Epoch: phi-1_5, Microsoft | open | Shots: 2 | 37.6% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 180 | bigcode/starcoder2-3bEpoch: starcoder2-3b, Hugging Face,ServiceNow,Nvidia,BigCode | open | default | 36.6% | not dated | Qwen2.5-Coder Technical Report |
| 181 | deepseek-ai/deepseek-coder-6.7b-baseEpoch: deepseek-coder-6.7b-base, DeepSeek,Peking University | open | default | 36.4% | not dated | Qwen2.5-Coder Technical Report |
| 182 | Salesforce/xgen-7b-8k-baseEpoch: xgen-7b-8k-base, Salesforce | open | Shots: 5 | 36.3% | not dated | XGen-7B Technical Report |
| 183 | LLaMA-7BMeta AIno page match | open | Shots: 5 | 35.6% | not dated | Qwen Technical Report |
| 184 | LLaMA-7BMeta AIno page match | open | Shots: 2 | 35.2% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 185 | LLaMA-7BMeta AIno page match | open | Shots: 5 | 35.1% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 186 | LLaMA-7BMeta AIno page match | open | Shots: 5 | 35.1% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 187 | LLaMA-7BMeta AIno page match | open | Shots: 5 | 35.1% | not dated | Baichuan 2: Open Large-scale Language Models |
| 188 | LLaMA-7BMeta AIno page match | open | Shots: 5 | 35.1% | not dated | XGen-7B Technical Report |
| 189 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 5 | 35.0% | not dated | Falcon2-11B Technical Report |
| 190 | Salesforce/xgen-7b-8k-baseEpoch: xgen-7b-8k-base, Salesforce | open | Shots: 0 | 32.1% | not dated | XGen-7B Technical Report |
| 191 | LLaMA-7BMeta AIno page match | open | Shots: 0 | 32.0% | not dated | XGen-7B Technical Report |
| 192 | mpt-7bMosaicMLno page match | open | Shots: 5 | 30.8% | not dated | Qwen Technical Report |
| 193 | open_llama_7b | not stated | Shots: 5 | 29.9% | not dated | XGen-7B Technical Report |
| 194 | open_llama_7b | not stated | Shots: 0 | 28.6% | not dated | XGen-7B Technical Report |
| 195 | Qwen/Qwen-1_8BEpoch: Qwen-1_8B | open | Shots: 5 | 28.2% | not dated | Qwen Technical Report |
| 196 | mpt-7bMosaicMLno page match | open | Shots: 5 | 27.9% | not dated | Baichuan 2: Open Large-scale Language Models |
| 197 | mpt-7bMosaicMLno page match | open | Shots: 0 | 27.4% | not dated | XGen-7B Technical Report |
| 198 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 2 | 26.9% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 199 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 5 | 26.9% | not dated | XGen-7B Technical Report |
| 200 | mpt-7bMosaicMLno page match | open | Shots: 5 | 26.8% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 201 | mpt-7bMosaicMLno page match | open | Shots: 2 | 26.8% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 202 | mpt-7bMosaicMLno page match | open | Shots: 5 | 26.7% | not dated | XGen-7B Technical Report |
| 203 | RedPajama-INCITE-7B-Base | not stated | Shots: 5 | 26.3% | not dated | XGen-7B Technical Report |
| 204 | Cerebras-GPT-13BCerebras Systemsno page match | open | Shots: 5 | 26.2% | not dated | XGen-7B Technical Report |
| 205 | dolly-v2-12bDatabricksno page match | open | Shots: 5 | 26.2% | not dated | XGen-7B Technical Report |
| 206 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 5 | 26.2% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 207 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 5 | 26.2% | not dated | Qwen Technical Report |
| 208 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 5 | 26.0% | not dated | Baichuan 2: Open Large-scale Language Models |
| 209 | deepseek-ai/deepseek-coder-1.3b-baseEpoch: deepseek-coder-1.3b-base, DeepSeek,Peking University | open | default | 25.8% | not dated | Qwen2.5-Coder Technical Report |
| 210 | RedPajama-INCITE-7B-Base | not stated | Shots: 0 | 25.8% | not dated | XGen-7B Technical Report |
| 211 | gpt-j-6bEleutherAI,LAIONno page match | open | Shots: 0 | 25.7% | not dated | XGen-7B Technical Report |
| 212 | dolly-v2-12bDatabricksno page match | open | Shots: 0 | 25.4% | not dated | XGen-7B Technical Report |
| 213 | gpt-j-6bEleutherAI,LAIONno page match | open | Shots: 5 | 25.1% | not dated | XGen-7B Technical Report |
| 214 | facebook/opt-13bEpoch: opt-13b | open | Shots: 5 | 25.1% | not dated | XGen-7B Technical Report |
| 215 | Cerebras-GPT-13BCerebras Systemsno page match | open | Shots: 0 | 24.6% | not dated | XGen-7B Technical Report |
| 216 | facebook/opt-13bEpoch: opt-13b | open | Shots: 0 | 24.4% | not dated | XGen-7B Technical Report |
| 217 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 0 | 23.9% | not dated | XGen-7B Technical Report |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method