Skip to content
Weights Not Walls
Menu

ARC AI2

Reported

Multiple-choice science questions from school exams (the AI2 Reasoning Challenge); this is the harder Challenge set.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Score shown
Epoch's column "Challenge score", as a percentage, higher is better; random guessing scores about 25.0%. The source gives no error bars.
Results
134 in all: 105 for open-weight models, 18 for closed ones, 11 where Epoch does not say. 40 linked to a model page here.

Best open-weight result: deepseek-ai/DeepSeek-V3, 95.3%, number 1 of 134 results.

ARC AI2 results
#ModelWeightsSettingScoreDateSource
1deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeekopenShots: 25-shot95.3%not datedDeepSeek-V3 Technical Report
2Llama-3.1-405BMeta AIno page matchopenShots: 25-shot95.3%not datedDeepSeek-V3 Technical Report
3Qwen2.5-72BAlibabano page matchopenShots: 25-shot94.5%not datedDeepSeek-V3 Technical Report
4DeepSeek-V2DeepSeekno page matchopenShots: 25-shot92.2%not datedDeepSeek-V3 Technical Report
5microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, MicrosoftopenShots: 10-shot91.6%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
6microsoft/Phi-3-small-8k-instructEpoch: Phi-3-small-8k-instruct, MicrosoftopenShots: 10-shot90.7%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
7gpt-3.5-turbo-1106OpenAIclosedShots: 10-shot87.4%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
8mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AIopenShots: 10-shot87.3%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
9claude-instant-1.2Anthropiccloseddefault86.3%not datedReleasing Claude Instant 1.2
10stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AIopenShots: 0-shot86.1%not datedQwen Technical Report
11claude-instant-1.1Anthropiccloseddefault85.7%not datedReleasing Claude Instant 1.2
12PaLM 540BGoogle ResearchclosedShots: 8-shot85.2%not datedGPT-4 Technical Report
13text-davinci-002OpenAIclosedShots: 25-shot85.2%not datedGPT-4 Technical Report
14microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, MicrosoftopenShots: 10-shot84.9%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
15Qwen-14BAlibabano page matchopenShots: 0-shot84.4%not datedQwen Technical Report
16Qwen-14BAlibabano page matchopenShots: 0-shot84.4%not datedNemotron-4 15B Technical Report
17meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AIopenShots: 10-shot82.8%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
18internlm/internlm-20bEpoch: internlm-20bopenShots: 0-shot81.7%not datedQwen Technical Report
19mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 10-shot78.6%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
20gemma-7bGoogle DeepMindno page matchopenShots: 10-shot78.3%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
21meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 0-shot78.3%not datedQwen Technical Report
22microsoft/phi-2Epoch: phi-2, MicrosoftopenShots: 10-shot75.9%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
23Qwen-7BAlibabano page matchopenShots: 0-shot75.3%not datedQwen Technical Report
24Qwen2.5-Coder-32BAlibabano page matchopendefault70.5%not datedQwen2.5-Coder Technical Report
25internlm/internlm-7bEpoch: internlm-7bopenShots: 0-shot69.5%not datedQwen Technical Report
26LLaMA-65BMeta AIno page matchopenShots: 0-shot69.5%not datedQwen Technical Report
27PaLM 2-Lnot statedShots: 2-shot69.2%not datedThe Falcon Series of Open Language Models
28PaLM 2-Lnot statedShots: 1-shot69.2%not datedPaLM 2 Technical Report
29falcon-180BTechnology Innovation Instituteno page matchopenShots: 2-shot67.8%not datedThe Falcon Series of Open Language Models
30LLaMA-33BMeta AIno page matchopenShots: 0-shot67.5%not datedQwen Technical Report
31Qwen/Qwen2.5-Coder-14BEpoch: Qwen2.5-Coder-14Bopendefault66.0%not datedQwen2.5-Coder Technical Report
32PaLM 2-Mnot statedShots: 2-shot64.9%not datedThe Falcon Series of Open Language Models
33PaLM 2-Mnot statedShots: 1-shot64.9%not datedPaLM 2 Technical Report
34deepseek-ai/DeepSeek-Coder-V2-BaseEpoch: DeepSeek-Coder-V2-Base, DeepSeekopendefault64.3%not datedQwen2.5-Coder Technical Report
35falcon-180BTechnology Innovation Instituteno page matchopendefault63.7%not datedThe Falcon Series of Open Language Models
36falcon-40bTechnology Innovation Instituteno page matchopenShots: 25-shot61.9%not datedFalcon2-11B Technical Report
37zai-org/chatglm2-6bEpoch: chatglm2-6bopenShots: 0-shot61.0%not datedQwen Technical Report
38Qwen2.5-Coder-7BAlibabano page matchopendefault60.9%not datedQwen2.5-Coder Technical Report
39Llama-2-13bMeta AIno page matchopenShots: 0-shot60.3%not datedQwen Technical Report
40PaLM 540BGoogle ResearchclosedShots: 2-shot60.1%not datedThe Falcon Series of Open Language Models
41PaLM 540BGoogle ResearchclosedShots: 1-shot60.1%not datedPaLM 2 Technical Report
42mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AIopenShots: 0-shot59.7%not datedMixtral of Experts
43PaLM 2-Snot statedShots: 2-shot59.6%not datedThe Falcon Series of Open Language Models
44PaLM 2-Snot statedShots: 1-shot59.6%not datedPaLM 2 Technical Report
45LLaMA-33BMeta AIno page matchopenShots: 0-shot57.8%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
46LLaMA-33BMeta AIno page matchopenShots: 0-shot57.8%not datedLLaMA: Open and Efficient Foundation Language Models
47meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopendefault57.4%not datedThe Falcon Series of Open Language Models
48meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 0-shot57.4%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
49deepseek-ai/DeepSeek-Coder-V2-Lite-BaseEpoch: DeepSeek-Coder-V2-Lite-Baseopendefault57.3%not datedQwen2.5-Coder Technical Report
50falcon-40bTechnology Innovation Instituteno page matchopendefault56.7%not datedThe Falcon Series of Open Language Models
51meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 0-shot56.5%not datedMixtral of Experts
52LLaMA-65BMeta AIno page matchopenShots: 0-shot56.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
53LLaMA-65BMeta AIno page matchopenShots: 0-shot56.0%not datedLLaMA: Open and Efficient Foundation Language Models
5401-ai/Yi-9BEpoch: Yi-9BopenShots: 25-shot55.6%not datedYi: Open Foundation Models by 01.AI
55mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 0-shot55.5%not datedNemotron-4 15B Technical Report
56Nemotron-4 15BNvidiaclosedShots: 0-shot55.5%not datedNemotron-4 15B Technical Report
57mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 0-shot54.9%not datedMixtral of Experts
58mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopendefault54.9%not datedGemma: Open Models Based on Gemini Research and Technology
59falcon-40bTechnology Innovation Instituteno page matchopenShots: 0-shot54.7%not datedFalcon2-11B Technical Report
60INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AInot stateddefault54.5%not datedINTELLECT-1 Technical Report
61falcon-40bTechnology Innovation Instituteno page matchopenShots: 0-shot54.5%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
62Llama-2-34bMeta AIcloseddefault54.5%not datedThe Falcon Series of Open Language Models
63Llama-2-34bMeta AIclosedShots: 0-shot54.5%not datedNemotron-4 15B Technical Report
64Llama-2-34bMeta AIclosedShots: 0-shot54.5%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
65LLaMA-33BMeta AIno page matchopenShots: 0-shot54.4%not datedMixtral of Experts
66gemma-7bGoogle DeepMindno page matchopenShots: 0-shot53.2%not datedNemotron-4 15B Technical Report
67gemma-7bGoogle DeepMindno page matchopendefault53.2%not datedGemma: Open Models Based on Gemini Research and Technology
68Qwen/Qwen-1_8BEpoch: Qwen-1_8BopenShots: 0-shot53.2%not datedQwen Technical Report
69text-davinci-001OpenAIcloseddefault53.2%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
70PaLM 540BGoogle Researchcloseddefault53.0%not datedThe Falcon Series of Open Language Models
71PaLM 540BGoogle ResearchclosedShots: 0-shot53.0%not datedLLaMA: Open and Efficient Foundation Language Models
72Qwen/Qwen2.5-Coder-3BEpoch: Qwen2.5-Coder-3Bopendefault52.9%not datedQwen2.5-Coder Technical Report
73LLaMA-13BMeta AIno page matchopenShots: 0-shot52.7%not datedQwen Technical Report
74LLaMA-13BMeta AIno page matchopenShots: 0-shot52.7%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
75LLaMA-13BMeta AIno page matchopenShots: 0-shot52.7%not datedLLaMA: Open and Efficient Foundation Language Models
76PaLM 62Bnot statedShots: 0-shot52.5%not datedLLaMA: Open and Efficient Foundation Language Models
77text-davinci-001OpenAIcloseddefault51.5%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
78text-davinci-001OpenAIcloseddefault51.4%not datedThe Falcon Series of Open Language Models
79text-davinci-001OpenAIclosedShots: 0-shot51.4%not datedLLaMA: Open and Efficient Foundation Language Models
80text-davinci-001OpenAIcloseddefault51.4%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
81mpt-30bMosaicMLno page matchopenShots: 0-shot50.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
82Yi-6B01.AIno page matchopenShots: 25-shot50.3%not datedYi: Open Foundation Models by 01.AI
83Llama-2-13bMeta AIno page matchopendefault49.4%not datedThe Falcon Series of Open Language Models
84Llama-2-13bMeta AIno page matchopenShots: 0-shot49.4%not datedNemotron-4 15B Technical Report
85Llama-2-13bMeta AIno page matchopenShots: 0-shot49.4%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
86Llama-2-13bMeta AIno page matchopendefault49.4%not datedGemma: Open Models Based on Gemini Research and Technology
87Llama-2-13bMeta AIno page matchopenShots: 0-shot48.8%not datedMixtral of Experts
88falcon-7bTechnology Innovation Instituteno page matchopenShots: 25-shot47.9%not datedFalcon2-11B Technical Report
89LLaMA-7BMeta AIno page matchopenShots: 0-shot47.6%not datedQwen Technical Report
90LLaMA-7BMeta AIno page matchopenShots: 0-shot47.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
91LLaMA-7BMeta AIno page matchopenShots: 0-shot47.6%not datedLLaMA: Open and Efficient Foundation Language Models
92bigcode/starcoder2-15bEpoch: starcoder2-15b, Hugging Face,ServiceNow,Nvidia,BigCodeopendefault47.2%not datedQwen2.5-Coder Technical Report
93Llama-2-7bMeta AIno page matchopendefault45.9%not datedThe Falcon Series of Open Language Models
94Llama-2-7bMeta AIno page matchopenShots: 0-shot45.9%not datedQwen Technical Report
95Llama-2-7bMeta AIno page matchopenShots: 0-shot45.9%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
96Llama-2-7bMeta AIno page matchopendefault45.9%not datedGemma: Open Models Based on Gemini Research and Technology
97Qwen2.5-Coder-1.5BAlibabano page matchopendefault45.2%not datedQwen2.5-Coder Technical Report
98LLaMA-7BMeta AIno page matchopendefault44.8%not datedXGen-7B Technical Report
99falcon-7bTechnology Innovation Instituteno page matchopendefault44.5%not datedThe Falcon Series of Open Language Models
100microsoft/phi-1_5Epoch: phi-1_5, Microsoftopendefault44.4%not datedTextbooks Are All You Need II: phi-1.5 technical report
101falcon-7bTechnology Innovation Instituteno page matchopenShots: 0-shot43.4%not datedFalcon2-11B Technical Report
102falcon-7bTechnology Innovation Instituteno page matchopendefault43.4%not datedXGen-7B Technical Report
103Llama-2-7bMeta AIno page matchopendefault43.4%not datedTextbooks Are All You Need II: phi-1.5 technical report
104Llama-2-7bMeta AIno page matchopenShots: 0-shot43.2%not datedMixtral of Experts
105vicuna-13b-v1.1not stateddefault43.2%not datedTextbooks Are All You Need II: phi-1.5 technical report
106mpt-7bMosaicMLno page matchopenShots: 0-shot42.6%not datedQwen Technical Report
107mpt-7bMosaicMLno page matchopenShots: 0-shot42.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
108falcon-7bTechnology Innovation Instituteno page matchopenShots: 0-shot42.4%not datedQwen Technical Report
109falcon-7bTechnology Innovation Instituteno page matchopenShots: 0-shot42.4%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
110deepseek-ai/deepseek-coder-33b-baseEpoch: deepseek-coder-33b-base, DeepSeek,Peking Universityopendefault42.2%not datedQwen2.5-Coder Technical Report
111gemma-2bGoogle DeepMindno page matchopendefault42.1%not datedGemma: Open Models Based on Gemini Research and Technology
112mpt-7bMosaicMLno page matchopendefault41.7%not datedXGen-7B Technical Report
113Salesforce/xgen-7b-8k-baseEpoch: xgen-7b-8k-base, Salesforceopendefault41.2%not datedXGen-7B Technical Report
114EleutherAI/gpt-neox-20bEpoch: gpt-neox-20b, EleutherAIopendefault41.1%not datedXGen-7B Technical Report
115mpt-7bMosaicMLno page matchopendefault40.5%not datedTextbooks Are All You Need II: phi-1.5 technical report
116dolly-v2-12bDatabricksno page matchopendefault39.6%not datedXGen-7B Technical Report
117RedPajama-INCITE-7B-Basenot stateddefault39.1%not datedXGen-7B Technical Report
118open_llama_7bnot stateddefault38.7%not datedXGen-7B Technical Report
119bigcode/starcoder2-7bEpoch: starcoder2-7b, Hugging Face,ServiceNow,Nvidia,BigCodeopendefault38.7%not datedQwen2.5-Coder Technical Report
120LLaMA-7BMeta AIno page matchopendefault38.5%not datedTextbooks Are All You Need II: phi-1.5 technical report
121Baichuan-2-13B-BaseBaichuanno page matchopenShots: 0-shot38.0%not datedQwen Technical Report
122deepseek-ai/deepseek-coder-6.7b-baseEpoch: deepseek-coder-6.7b-base, DeepSeek,Peking Universityopendefault36.4%not datedQwen2.5-Coder Technical Report
123falcon-7bTechnology Innovation Instituteno page matchopendefault36.3%not datedTextbooks Are All You Need II: phi-1.5 technical report
124gpt-j-6bEleutherAI,LAIONno page matchopendefault36.3%not datedXGen-7B Technical Report
125facebook/opt-13bEpoch: opt-13bopendefault35.8%not datedXGen-7B Technical Report
126Qwen/CodeQwen1.5-7BEpoch: CodeQwen1.5-7Bopendefault35.7%not datedQwen2.5-Coder Technical Report
127Qwen/Qwen2.5-Coder-0.5BEpoch: Qwen2.5-Coder-0.5Bopendefault34.4%not datedQwen2.5-Coder Technical Report
128bigcode/starcoder2-3bEpoch: starcoder2-3b, Hugging Face,ServiceNow,Nvidia,BigCodeopendefault34.2%not datedQwen2.5-Coder Technical Report
129Baichuan-2-7B-BaseBaichuanno page matchopenShots: 0-shot32.5%not datedQwen Technical Report
130Cerebras-GPT-13BCerebras Systemsno page matchopendefault32.4%not datedXGen-7B Technical Report
131stabilityai/stablelm-tuned-alpha-7bEpoch: stablelm-tuned-alpha-7bopendefault27.0%not datedXGen-7B Technical Report
132deepseek-ai/deepseek-coder-1.3b-baseEpoch: deepseek-coder-1.3b-base, DeepSeek,Peking Universityopendefault25.4%not datedQwen2.5-Coder Technical Report
133gpt2-xlOpenAIno page matchopendefault25.0%not datedTextbooks Are All You Need II: phi-1.5 technical report
134facebook/opt-1.3bEpoch: opt-1.3b, Meta AIopendefault23.2%not datedTextbooks Are All You Need II: phi-1.5 technical report

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method