ARC AI2
ReportedMultiple-choice science questions from school exams (the AI2 Reasoning Challenge); this is the harder Challenge set.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Score shown
- Epoch's column "Challenge score", as a percentage, higher is better; random guessing scores about 25.0%. The source gives no error bars.
- Results
- 134 in all: 105 for open-weight models, 18 for closed ones, 11 where Epoch does not say. 40 linked to a model page here.
Best open-weight result: deepseek-ai/DeepSeek-V3, 95.3%, number 1 of 134 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeek | open | Shots: 25-shot | 95.3% | not dated | DeepSeek-V3 Technical Report |
| 2 | Llama-3.1-405BMeta AIno page match | open | Shots: 25-shot | 95.3% | not dated | DeepSeek-V3 Technical Report |
| 3 | Qwen2.5-72BAlibabano page match | open | Shots: 25-shot | 94.5% | not dated | DeepSeek-V3 Technical Report |
| 4 | DeepSeek-V2DeepSeekno page match | open | Shots: 25-shot | 92.2% | not dated | DeepSeek-V3 Technical Report |
| 5 | microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, Microsoft | open | Shots: 10-shot | 91.6% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 6 | microsoft/Phi-3-small-8k-instructEpoch: Phi-3-small-8k-instruct, Microsoft | open | Shots: 10-shot | 90.7% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 7 | gpt-3.5-turbo-1106OpenAI | closed | Shots: 10-shot | 87.4% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 8 | mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AI | open | Shots: 10-shot | 87.3% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 9 | claude-instant-1.2Anthropic | closed | default | 86.3% | not dated | Releasing Claude Instant 1.2 |
| 10 | stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AI | open | Shots: 0-shot | 86.1% | not dated | Qwen Technical Report |
| 11 | claude-instant-1.1Anthropic | closed | default | 85.7% | not dated | Releasing Claude Instant 1.2 |
| 12 | PaLM 540BGoogle Research | closed | Shots: 8-shot | 85.2% | not dated | GPT-4 Technical Report |
| 13 | text-davinci-002OpenAI | closed | Shots: 25-shot | 85.2% | not dated | GPT-4 Technical Report |
| 14 | microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, Microsoft | open | Shots: 10-shot | 84.9% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 15 | Qwen-14BAlibabano page match | open | Shots: 0-shot | 84.4% | not dated | Qwen Technical Report |
| 16 | Qwen-14BAlibabano page match | open | Shots: 0-shot | 84.4% | not dated | Nemotron-4 15B Technical Report |
| 17 | meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AI | open | Shots: 10-shot | 82.8% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 18 | internlm/internlm-20bEpoch: internlm-20b | open | Shots: 0-shot | 81.7% | not dated | Qwen Technical Report |
| 19 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 10-shot | 78.6% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 20 | gemma-7bGoogle DeepMindno page match | open | Shots: 10-shot | 78.3% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 21 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 0-shot | 78.3% | not dated | Qwen Technical Report |
| 22 | microsoft/phi-2Epoch: phi-2, Microsoft | open | Shots: 10-shot | 75.9% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 23 | Qwen-7BAlibabano page match | open | Shots: 0-shot | 75.3% | not dated | Qwen Technical Report |
| 24 | Qwen2.5-Coder-32BAlibabano page match | open | default | 70.5% | not dated | Qwen2.5-Coder Technical Report |
| 25 | internlm/internlm-7bEpoch: internlm-7b | open | Shots: 0-shot | 69.5% | not dated | Qwen Technical Report |
| 26 | LLaMA-65BMeta AIno page match | open | Shots: 0-shot | 69.5% | not dated | Qwen Technical Report |
| 27 | PaLM 2-L | not stated | Shots: 2-shot | 69.2% | not dated | The Falcon Series of Open Language Models |
| 28 | PaLM 2-L | not stated | Shots: 1-shot | 69.2% | not dated | PaLM 2 Technical Report |
| 29 | falcon-180BTechnology Innovation Instituteno page match | open | Shots: 2-shot | 67.8% | not dated | The Falcon Series of Open Language Models |
| 30 | LLaMA-33BMeta AIno page match | open | Shots: 0-shot | 67.5% | not dated | Qwen Technical Report |
| 31 | Qwen/Qwen2.5-Coder-14BEpoch: Qwen2.5-Coder-14B | open | default | 66.0% | not dated | Qwen2.5-Coder Technical Report |
| 32 | PaLM 2-M | not stated | Shots: 2-shot | 64.9% | not dated | The Falcon Series of Open Language Models |
| 33 | PaLM 2-M | not stated | Shots: 1-shot | 64.9% | not dated | PaLM 2 Technical Report |
| 34 | deepseek-ai/DeepSeek-Coder-V2-BaseEpoch: DeepSeek-Coder-V2-Base, DeepSeek | open | default | 64.3% | not dated | Qwen2.5-Coder Technical Report |
| 35 | falcon-180BTechnology Innovation Instituteno page match | open | default | 63.7% | not dated | The Falcon Series of Open Language Models |
| 36 | falcon-40bTechnology Innovation Instituteno page match | open | Shots: 25-shot | 61.9% | not dated | Falcon2-11B Technical Report |
| 37 | zai-org/chatglm2-6bEpoch: chatglm2-6b | open | Shots: 0-shot | 61.0% | not dated | Qwen Technical Report |
| 38 | Qwen2.5-Coder-7BAlibabano page match | open | default | 60.9% | not dated | Qwen2.5-Coder Technical Report |
| 39 | Llama-2-13bMeta AIno page match | open | Shots: 0-shot | 60.3% | not dated | Qwen Technical Report |
| 40 | PaLM 540BGoogle Research | closed | Shots: 2-shot | 60.1% | not dated | The Falcon Series of Open Language Models |
| 41 | PaLM 540BGoogle Research | closed | Shots: 1-shot | 60.1% | not dated | PaLM 2 Technical Report |
| 42 | mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AI | open | Shots: 0-shot | 59.7% | not dated | Mixtral of Experts |
| 43 | PaLM 2-S | not stated | Shots: 2-shot | 59.6% | not dated | The Falcon Series of Open Language Models |
| 44 | PaLM 2-S | not stated | Shots: 1-shot | 59.6% | not dated | PaLM 2 Technical Report |
| 45 | LLaMA-33BMeta AIno page match | open | Shots: 0-shot | 57.8% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 46 | LLaMA-33BMeta AIno page match | open | Shots: 0-shot | 57.8% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 47 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | default | 57.4% | not dated | The Falcon Series of Open Language Models |
| 48 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 0-shot | 57.4% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 49 | deepseek-ai/DeepSeek-Coder-V2-Lite-BaseEpoch: DeepSeek-Coder-V2-Lite-Base | open | default | 57.3% | not dated | Qwen2.5-Coder Technical Report |
| 50 | falcon-40bTechnology Innovation Instituteno page match | open | default | 56.7% | not dated | The Falcon Series of Open Language Models |
| 51 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 0-shot | 56.5% | not dated | Mixtral of Experts |
| 52 | LLaMA-65BMeta AIno page match | open | Shots: 0-shot | 56.0% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 53 | LLaMA-65BMeta AIno page match | open | Shots: 0-shot | 56.0% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 54 | 01-ai/Yi-9BEpoch: Yi-9B | open | Shots: 25-shot | 55.6% | not dated | Yi: Open Foundation Models by 01.AI |
| 55 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 0-shot | 55.5% | not dated | Nemotron-4 15B Technical Report |
| 56 | Nemotron-4 15BNvidia | closed | Shots: 0-shot | 55.5% | not dated | Nemotron-4 15B Technical Report |
| 57 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 0-shot | 54.9% | not dated | Mixtral of Experts |
| 58 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | default | 54.9% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 59 | falcon-40bTechnology Innovation Instituteno page match | open | Shots: 0-shot | 54.7% | not dated | Falcon2-11B Technical Report |
| 60 | INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AI | not stated | default | 54.5% | not dated | INTELLECT-1 Technical Report |
| 61 | falcon-40bTechnology Innovation Instituteno page match | open | Shots: 0-shot | 54.5% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 62 | Llama-2-34bMeta AI | closed | default | 54.5% | not dated | The Falcon Series of Open Language Models |
| 63 | Llama-2-34bMeta AI | closed | Shots: 0-shot | 54.5% | not dated | Nemotron-4 15B Technical Report |
| 64 | Llama-2-34bMeta AI | closed | Shots: 0-shot | 54.5% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 65 | LLaMA-33BMeta AIno page match | open | Shots: 0-shot | 54.4% | not dated | Mixtral of Experts |
| 66 | gemma-7bGoogle DeepMindno page match | open | Shots: 0-shot | 53.2% | not dated | Nemotron-4 15B Technical Report |
| 67 | gemma-7bGoogle DeepMindno page match | open | default | 53.2% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 68 | Qwen/Qwen-1_8BEpoch: Qwen-1_8B | open | Shots: 0-shot | 53.2% | not dated | Qwen Technical Report |
| 69 | text-davinci-001OpenAI | closed | default | 53.2% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 70 | PaLM 540BGoogle Research | closed | default | 53.0% | not dated | The Falcon Series of Open Language Models |
| 71 | PaLM 540BGoogle Research | closed | Shots: 0-shot | 53.0% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 72 | Qwen/Qwen2.5-Coder-3BEpoch: Qwen2.5-Coder-3B | open | default | 52.9% | not dated | Qwen2.5-Coder Technical Report |
| 73 | LLaMA-13BMeta AIno page match | open | Shots: 0-shot | 52.7% | not dated | Qwen Technical Report |
| 74 | LLaMA-13BMeta AIno page match | open | Shots: 0-shot | 52.7% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 75 | LLaMA-13BMeta AIno page match | open | Shots: 0-shot | 52.7% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 76 | PaLM 62B | not stated | Shots: 0-shot | 52.5% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 77 | text-davinci-001OpenAI | closed | default | 51.5% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 78 | text-davinci-001OpenAI | closed | default | 51.4% | not dated | The Falcon Series of Open Language Models |
| 79 | text-davinci-001OpenAI | closed | Shots: 0-shot | 51.4% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 80 | text-davinci-001OpenAI | closed | default | 51.4% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 81 | mpt-30bMosaicMLno page match | open | Shots: 0-shot | 50.6% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 82 | Yi-6B01.AIno page match | open | Shots: 25-shot | 50.3% | not dated | Yi: Open Foundation Models by 01.AI |
| 83 | Llama-2-13bMeta AIno page match | open | default | 49.4% | not dated | The Falcon Series of Open Language Models |
| 84 | Llama-2-13bMeta AIno page match | open | Shots: 0-shot | 49.4% | not dated | Nemotron-4 15B Technical Report |
| 85 | Llama-2-13bMeta AIno page match | open | Shots: 0-shot | 49.4% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 86 | Llama-2-13bMeta AIno page match | open | default | 49.4% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 87 | Llama-2-13bMeta AIno page match | open | Shots: 0-shot | 48.8% | not dated | Mixtral of Experts |
| 88 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 25-shot | 47.9% | not dated | Falcon2-11B Technical Report |
| 89 | LLaMA-7BMeta AIno page match | open | Shots: 0-shot | 47.6% | not dated | Qwen Technical Report |
| 90 | LLaMA-7BMeta AIno page match | open | Shots: 0-shot | 47.6% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 91 | LLaMA-7BMeta AIno page match | open | Shots: 0-shot | 47.6% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 92 | bigcode/starcoder2-15bEpoch: starcoder2-15b, Hugging Face,ServiceNow,Nvidia,BigCode | open | default | 47.2% | not dated | Qwen2.5-Coder Technical Report |
| 93 | Llama-2-7bMeta AIno page match | open | default | 45.9% | not dated | The Falcon Series of Open Language Models |
| 94 | Llama-2-7bMeta AIno page match | open | Shots: 0-shot | 45.9% | not dated | Qwen Technical Report |
| 95 | Llama-2-7bMeta AIno page match | open | Shots: 0-shot | 45.9% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 96 | Llama-2-7bMeta AIno page match | open | default | 45.9% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 97 | Qwen2.5-Coder-1.5BAlibabano page match | open | default | 45.2% | not dated | Qwen2.5-Coder Technical Report |
| 98 | LLaMA-7BMeta AIno page match | open | default | 44.8% | not dated | XGen-7B Technical Report |
| 99 | falcon-7bTechnology Innovation Instituteno page match | open | default | 44.5% | not dated | The Falcon Series of Open Language Models |
| 100 | microsoft/phi-1_5Epoch: phi-1_5, Microsoft | open | default | 44.4% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 101 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 0-shot | 43.4% | not dated | Falcon2-11B Technical Report |
| 102 | falcon-7bTechnology Innovation Instituteno page match | open | default | 43.4% | not dated | XGen-7B Technical Report |
| 103 | Llama-2-7bMeta AIno page match | open | default | 43.4% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 104 | Llama-2-7bMeta AIno page match | open | Shots: 0-shot | 43.2% | not dated | Mixtral of Experts |
| 105 | vicuna-13b-v1.1 | not stated | default | 43.2% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 106 | mpt-7bMosaicMLno page match | open | Shots: 0-shot | 42.6% | not dated | Qwen Technical Report |
| 107 | mpt-7bMosaicMLno page match | open | Shots: 0-shot | 42.6% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 108 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 0-shot | 42.4% | not dated | Qwen Technical Report |
| 109 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 0-shot | 42.4% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 110 | deepseek-ai/deepseek-coder-33b-baseEpoch: deepseek-coder-33b-base, DeepSeek,Peking University | open | default | 42.2% | not dated | Qwen2.5-Coder Technical Report |
| 111 | gemma-2bGoogle DeepMindno page match | open | default | 42.1% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 112 | mpt-7bMosaicMLno page match | open | default | 41.7% | not dated | XGen-7B Technical Report |
| 113 | Salesforce/xgen-7b-8k-baseEpoch: xgen-7b-8k-base, Salesforce | open | default | 41.2% | not dated | XGen-7B Technical Report |
| 114 | EleutherAI/gpt-neox-20bEpoch: gpt-neox-20b, EleutherAI | open | default | 41.1% | not dated | XGen-7B Technical Report |
| 115 | mpt-7bMosaicMLno page match | open | default | 40.5% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 116 | dolly-v2-12bDatabricksno page match | open | default | 39.6% | not dated | XGen-7B Technical Report |
| 117 | RedPajama-INCITE-7B-Base | not stated | default | 39.1% | not dated | XGen-7B Technical Report |
| 118 | open_llama_7b | not stated | default | 38.7% | not dated | XGen-7B Technical Report |
| 119 | bigcode/starcoder2-7bEpoch: starcoder2-7b, Hugging Face,ServiceNow,Nvidia,BigCode | open | default | 38.7% | not dated | Qwen2.5-Coder Technical Report |
| 120 | LLaMA-7BMeta AIno page match | open | default | 38.5% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 121 | Baichuan-2-13B-BaseBaichuanno page match | open | Shots: 0-shot | 38.0% | not dated | Qwen Technical Report |
| 122 | deepseek-ai/deepseek-coder-6.7b-baseEpoch: deepseek-coder-6.7b-base, DeepSeek,Peking University | open | default | 36.4% | not dated | Qwen2.5-Coder Technical Report |
| 123 | falcon-7bTechnology Innovation Instituteno page match | open | default | 36.3% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 124 | gpt-j-6bEleutherAI,LAIONno page match | open | default | 36.3% | not dated | XGen-7B Technical Report |
| 125 | facebook/opt-13bEpoch: opt-13b | open | default | 35.8% | not dated | XGen-7B Technical Report |
| 126 | Qwen/CodeQwen1.5-7BEpoch: CodeQwen1.5-7B | open | default | 35.7% | not dated | Qwen2.5-Coder Technical Report |
| 127 | Qwen/Qwen2.5-Coder-0.5BEpoch: Qwen2.5-Coder-0.5B | open | default | 34.4% | not dated | Qwen2.5-Coder Technical Report |
| 128 | bigcode/starcoder2-3bEpoch: starcoder2-3b, Hugging Face,ServiceNow,Nvidia,BigCode | open | default | 34.2% | not dated | Qwen2.5-Coder Technical Report |
| 129 | Baichuan-2-7B-BaseBaichuanno page match | open | Shots: 0-shot | 32.5% | not dated | Qwen Technical Report |
| 130 | Cerebras-GPT-13BCerebras Systemsno page match | open | default | 32.4% | not dated | XGen-7B Technical Report |
| 131 | stabilityai/stablelm-tuned-alpha-7bEpoch: stablelm-tuned-alpha-7b | open | default | 27.0% | not dated | XGen-7B Technical Report |
| 132 | deepseek-ai/deepseek-coder-1.3b-baseEpoch: deepseek-coder-1.3b-base, DeepSeek,Peking University | open | default | 25.4% | not dated | Qwen2.5-Coder Technical Report |
| 133 | gpt2-xlOpenAIno page match | open | default | 25.0% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 134 | facebook/opt-1.3bEpoch: opt-1.3b, Meta AI | open | default | 23.2% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method