Skip to content
Weights Not Walls
Menu

GSM8K

Reported

Grade-school mathematics word problems that need a few steps of arithmetic.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Score shown
Epoch's column "EM", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
Results
170 in all: 140 for open-weight models, 19 for closed ones, 11 where Epoch does not say. 63 linked to a model page here.

Best open-weight result: deepseek-ai/DeepSeek-Coder-V2-Instruct, 94.5%, number 1 of 170 results.

GSM8K results
#ModelWeightsSettingScoreDateSource
1deepseek-ai/DeepSeek-Coder-V2-InstructEpoch: DeepSeek-Coder-V2-Instruct, DeepSeekopendefault94.5%not datedQwen2.5-Coder Technical Report
2Qwen/Qwen2.5-Coder-14B-InstructEpoch: Qwen2.5-Coder-14B-Instructopendefault94.2%not datedQwen2.5-Coder Technical Report
3Qwen/Qwen2.5-Coder-32B-InstructEpoch: Qwen2.5-Coder-32B-Instruct, Alibabaopendefault93.0%not datedQwen2.5-Coder Technical Report
4gpt-4-0314OpenAIclosedShots: 592.0%not datedGPT-4 Technical Report
5gpt-4o-mini-2024-07-18OpenAIclosedShots: 891.3%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
6Qwen2.5-Coder-32BAlibabano page matchopenShots: 491.1%not datedQwen2.5-Coder Technical Report
7gpt-4-0613OpenAIclosedShots: 490.0%not datedBaichuan 2: Open Large-scale Language Models
8microsoft/Phi-3.5-MoE-instructEpoch: Phi-3.5-MoE-instruct, MicrosoftopenShots: 888.7%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
9Qwen/Qwen2.5-Coder-14BEpoch: Qwen2.5-Coder-14BopenShots: 488.7%not datedQwen2.5-Coder Technical Report
10deepseek-ai/DeepSeek-Coder-V2-Lite-InstructEpoch: DeepSeek-Coder-V2-Lite-Instructopendefault87.6%not datedQwen2.5-Coder Technical Report
11claude-instant-1.2AnthropicclosedShots: 086.7%not datedReleasing Claude Instant 1.2
12Qwen/Qwen2.5-Coder-7B-InstructEpoch: Qwen2.5-Coder-7B-Instruct, Alibabaopendefault86.7%not datedQwen2.5-Coder Technical Report
13microsoft/Phi-3.5-mini-instructEpoch: Phi-3.5-mini-instruct, MicrosoftopenShots: 886.2%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
14deepseek-ai/DeepSeek-Coder-V2-BaseEpoch: DeepSeek-Coder-V2-Base, DeepSeekopenShots: 485.8%not datedQwen2.5-Coder Technical Report
15gemma-2-9bGoogle DeepMindno page matchopenShots: 884.9%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
16mistralai/Mistral-Nemo-Base-2407Epoch: Mistral-Nemo-Base-2407, Mistral AIopenShots: 884.2%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
17Qwen2.5-Coder-7BAlibabano page matchopenShots: 483.9%not datedQwen2.5-Coder Technical Report
18gemini-1.5-flash-001Google DeepMindclosedShots: 882.4%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
19meta-llama/Llama-3.1-8B-InstructEpoch: Llama-3.1-8B-Instruct, Meta AIopenShots: 882.4%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
20claude-instant-1.1AnthropicclosedShots: 080.9%not datedReleasing Claude Instant 1.2
21Qwen/Qwen2.5-Coder-3B-InstructEpoch: Qwen2.5-Coder-3B-Instructopendefault80.7%not datedQwen2.5-Coder Technical Report
22text-davinci-003OpenAInot statedShots: 878.2%not datedQwen Technical Report
2301-ai/Yi-34B-ChatEpoch: Yi-34B-Chat, 01.AIopenShots: 476.0%not datedYi: Open Foundation Models by 01.AI
24Qwen/Qwen2.5-Coder-3BEpoch: Qwen2.5-Coder-3BopenShots: 475.7%not datedQwen2.5-Coder Technical Report
25mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AIopenShots: 874.4%not datedMixtral of Experts
2601-ai/Yi-34B-ChatEpoch: Yi-34B-Chat, 01.AIopenShots: 071.7%not datedYi: Open Foundation Models by 01.AI
27meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 869.6%not datedMixtral of Experts
28stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AIopenShots: 869.6%not datedQwen Technical Report
29Yi-34B01.AIno page matchopendefault67.2%not datedYi: Open Foundation Models by 01.AI
30deepseek-ai/DeepSeek-Coder-V2-Lite-BaseEpoch: DeepSeek-Coder-V2-Lite-BaseopenShots: 467.1%not datedQwen2.5-Coder Technical Report
31Qwen2.5-Coder-1.5BAlibabano page matchopenShots: 465.8%not datedQwen2.5-Coder Technical Report
32meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 863.3%not datedQwen Technical Report
33internlm/internlm-20bEpoch: internlm-20bopendefault62.9%not datedYi: Open Foundation Models by 01.AI
34Qwen-14BAlibabano page matchopendefault61.3%not datedYi: Open Foundation Models by 01.AI
35Qwen-14BAlibabano page matchopenShots: 861.3%not datedQwen Technical Report
36Qwen/Qwen-14B-ChatEpoch: Qwen-14B-Chat, AlibabaopenShots: 461.2%not datedYi: Open Foundation Models by 01.AI
37Qwen-14BAlibabano page matchopenShots: 860.1%not datedNemotron-4 15B Technical Report
38Qwen/Qwen-14B-ChatEpoch: Qwen-14B-Chat, AlibabaopenShots: 059.5%not datedYi: Open Foundation Models by 01.AI
39meta-llama/Llama-2-70b-chatEpoch: Llama-2-70b-chat, Meta AIopenShots: 458.7%not datedYi: Open Foundation Models by 01.AI
40gpt-3.5-turbo-0613OpenAIclosedShots: 457.8%not datedBaichuan 2: Open Large-scale Language Models
41bigcode/starcoder2-15bEpoch: starcoder2-15b, Hugging Face,ServiceNow,Nvidia,BigCodeopenShots: 457.7%not datedQwen2.5-Coder Technical Report
42text-davinci-003OpenAInot stateddefault57.1%not datedYi: Open Foundation Models by 01.AI
43text-davinci-003OpenAInot statedShots: 557.1%not datedGPT-4 Technical Report
44code-davinci-002OpenAIclosedShots: 556.8%not datedStanford HELM
45meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopendefault56.8%not datedYi: Open Foundation Models by 01.AI
46meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 856.8%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
47PaLM 540BGoogle Researchcloseddefault56.5%not datedLLaMA: Open and Efficient Foundation Language Models
48falcon-180BTechnology Innovation Instituteno page matchopendefault54.4%not datedYi: Open Foundation Models by 01.AI
49LLaMA-65BMeta AIno page matchopenShots: 854.4%not datedQwen Technical Report
50mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 854.4%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
51tiiuae/falcon-11BEpoch: falcon-11b, Technology Innovation InstituteopenShots: 553.8%not datedFalcon2-11B Technical Report
52Baichuan-2-13B-BaseBaichuanno page matchopenShots: 852.8%not datedNemotron-4 15B Technical Report
53Baichuan-2-13B-BaseBaichuanno page matchopenShots: 452.8%not datedBaichuan 2: Open Large-scale Language Models
54Baichuan-2-13B-BaseBaichuanno page matchopenShots: 852.8%not datedQwen Technical Report
55internlm/internlm-20bEpoch: internlm-20bopenShots: 852.6%not datedQwen Technical Report
56Qwen-7BAlibabano page matchopendefault51.7%not datedYi: Open Foundation Models by 01.AI
57Qwen-7BAlibabano page matchopenShots: 851.7%not datedQwen Technical Report
58LLaMA-65BMeta AIno page matchopenShots: 850.9%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
59LLaMA-65BMeta AIno page matchopendefault50.9%not datedLLaMA: Open and Efficient Foundation Language Models
60LLaMA-65BMeta AIno page matchopenShots: 050.9%not datedTextbooks Are All You Need II: phi-1.5 technical report
61text-davinci-003OpenAInot statedShots: 550.6%not datedStanford HELM
62mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 850.0%not datedMixtral of Experts
63meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 548.4%not datedStanford HELM
64mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopendefault47.5%not datedYi: Open Foundation Models by 01.AI
65meta-llama/Llama-2-70b-chatEpoch: Llama-2-70b-chat, Meta AIopenShots: 047.1%not datedYi: Open Foundation Models by 01.AI
66gpt-3.5-turbo-0613OpenAIclosedShots: 546.9%not datedStanford HELM
67LLaMA-65BMeta AIno page matchopenShots: 546.6%not datedStanford HELM
68gemma-7bGoogle DeepMindno page matchopendefault46.4%not datedGemma: Open Models Based on Gemini Research and Technology
69gemma-7bGoogle DeepMindno page matchopenShots: 846.4%not datedNemotron-4 15B Technical Report
70Nemotron-4 15BNvidiaclosedShots: 846.0%not datedNemotron-4 15B Technical Report
71baichuan-inc/Baichuan2-13B-ChatEpoch: Baichuan2-13B-ChatopenShots: 045.7%not datedYi: Open Foundation Models by 01.AI
7201-ai/Yi-6B-ChatEpoch: Yi-6B-Chat, 01.AIopenShots: 444.9%not datedYi: Open Foundation Models by 01.AI
73LLaMA-33BMeta AIno page matchopenShots: 844.1%not datedMixtral of Experts
74internlm/internlm-20bEpoch: internlm-20bopenShots: 443.4%not datedYi: Open Foundation Models by 01.AI
75LLaMA-33BMeta AIno page matchopenShots: 842.3%not datedQwen Technical Report
76Llama-2-34bMeta AIclosedShots: 842.2%not datedNemotron-4 15B Technical Report
77Llama-2-34bMeta AIcloseddefault42.2%not datedYi: Open Foundation Models by 01.AI
78Llama-2-34bMeta AIclosedShots: 842.2%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
79Llama-2-34bMeta AIclosedShots: 842.2%not datedQwen Technical Report
80text-davinci-002OpenAIclosedShots: 541.5%not datedStanford HELM
81INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AInot statedShots: 538.6%not datedINTELLECT-1 Technical Report
8201-ai/Yi-6B-ChatEpoch: Yi-6B-Chat, 01.AIopenShots: 038.4%not datedYi: Open Foundation Models by 01.AI
83mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 538.1%not datedStanford HELM
84Qwen/CodeQwen1.5-7BEpoch: CodeQwen1.5-7BopenShots: 437.7%not datedQwen2.5-Coder Technical Report
85meta-llama/Llama-2-13b-chatEpoch: Llama-2-13b-chat, Meta AIopenShots: 036.9%not datedYi: Open Foundation Models by 01.AI
86LLaMA-33BMeta AIno page matchopenShots: 835.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
87LLaMA-33BMeta AIno page matchopendefault35.6%not datedLLaMA: Open and Efficient Foundation Language Models
88deepseek-ai/deepseek-coder-33b-baseEpoch: deepseek-coder-33b-base, DeepSeek,Peking UniversityopenShots: 435.4%not datedQwen2.5-Coder Technical Report
89mistralai/Mistral-7B-Instruct-v0.2Epoch: Mistral-7B-Instruct-v0.2, Mistral AIopendefault35.4%not datedGemma: Open Models Based on Gemini Research and Technology
90mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 835.4%not datedNemotron-4 15B Technical Report
91Qwen/Qwen2.5-Coder-0.5BEpoch: Qwen2.5-Coder-0.5BopenShots: 434.5%not datedQwen2.5-Coder Technical Report
92mpt-30b-instructMosaicMLno page matchopenShots: 534.4%not datedStanford HELM
93Llama-2-13bMeta AIno page matchopenShots: 834.3%not datedMixtral of Experts
94tiiuae/falcon-40b-instructEpoch: falcon-40b-instruct, Technology Innovation InstituteopenShots: 533.8%not datedStanford HELM
95PaLM 62Bnot stateddefault33.0%not datedLLaMA: Open and Efficient Foundation Language Models
96bigcode/starcoder2-7bEpoch: starcoder2-7b, Hugging Face,ServiceNow,Nvidia,BigCodeopenShots: 432.7%not datedQwen2.5-Coder Technical Report
97Yi-6B01.AIno page matchopendefault32.5%not datedYi: Open Foundation Models by 01.AI
98zai-org/chatglm2-6bEpoch: chatglm2-6bopenShots: 832.4%not datedQwen Technical Report
99LLaMA-33BMeta AIno page matchopenShots: 532.0%not datedStanford HELM
100internlm/internlm-7bEpoch: internlm-7bopenShots: 831.2%not datedQwen Technical Report
101Llama-2-13bMeta AIno page matchopenShots: 829.6%not datedQwen Technical Report
102zai-org/chatglm2-6bEpoch: chatglm2-6bopenShots: 428.9%not datedBaichuan 2: Open Large-scale Language Models
103Llama-2-13bMeta AIno page matchopenShots: 428.9%not datedBaichuan 2: Open Large-scale Language Models
104zai-org/chatglm2-6bEpoch: chatglm2-6bopenShots: 828.8%not datedQwen Technical Report
105Llama-2-13bMeta AIno page matchopendefault28.7%not datedGemma: Open Models Based on Gemini Research and Technology
106Llama-2-13bMeta AIno page matchopenShots: 828.7%not datedNemotron-4 15B Technical Report
107Llama-2-13bMeta AIno page matchopenShots: 828.7%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
108vicuna-13b-v1.1not statedShots: 428.1%not datedBaichuan 2: Open Large-scale Language Models
109baichuan-inc/Baichuan-13B-BaseEpoch: Baichuan-13B-Base, BaichuanopenShots: 426.8%not datedBaichuan 2: Open Large-scale Language Models
110falcon-40bTechnology Innovation Instituteno page matchopenShots: 525.0%not datedStanford HELM
111Baichuan-2-7B-BaseBaichuanno page matchopenShots: 824.6%not datedQwen Technical Report
112Baichuan-2-7B-BaseBaichuanno page matchopenShots: 424.5%not datedBaichuan 2: Open Large-scale Language Models
113Baichuan-2-7B-BaseBaichuanno page matchopendefault24.5%not datedYi: Open Foundation Models by 01.AI
114Llama-2-13bMeta AIno page matchopenShots: 524.5%not datedStanford HELM
115baichuan-inc/Baichuan2-13B-ChatEpoch: Baichuan2-13B-ChatopenShots: 423.3%not datedYi: Open Foundation Models by 01.AI
116vicuna-13b-v1.3Large Model Systems Organization,University of California (UC) Berkeleyno page matchopenShots: 522.6%not datedStanford HELM
117Baichuan-2-13B-BaseBaichuanno page matchopendefault22.1%not datedYi: Open Foundation Models by 01.AI
118bigcode/starcoder2-3bEpoch: starcoder2-3b, Hugging Face,ServiceNow,Nvidia,BigCodeopenShots: 421.6%not datedQwen2.5-Coder Technical Report
119falcon-40bTechnology Innovation Instituteno page matchopenShots: 521.5%not datedFalcon2-11B Technical Report
120deepseek-ai/deepseek-coder-6.7b-baseEpoch: deepseek-coder-6.7b-base, DeepSeek,Peking UniversityopenShots: 421.3%not datedQwen2.5-Coder Technical Report
121Qwen/Qwen-1_8BEpoch: Qwen-1_8BopenShots: 821.2%not datedQwen Technical Report
122LLaMA-13BMeta AIno page matchopenShots: 420.5%not datedBaichuan 2: Open Large-scale Language Models
123LLaMA-13BMeta AIno page matchopenShots: 820.3%not datedQwen Technical Report
124falcon-40bTechnology Innovation Instituteno page matchopenShots: 819.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
125falcon-40bTechnology Innovation Instituteno page matchopenShots: 819.6%not datedQwen Technical Report
126LLaMA-13BMeta AIno page matchopenShots: 817.8%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
127LLaMA-13BMeta AIno page matchopendefault17.8%not datedLLaMA: Open and Efficient Foundation Language Models
128gemma-2bGoogle DeepMindno page matchopendefault17.7%not datedGemma: Open Models Based on Gemini Research and Technology
129Llama-2-7bMeta AIno page matchopendefault16.7%not datedYi: Open Foundation Models by 01.AI
130Llama-2-7bMeta AIno page matchopenShots: 816.7%not datedQwen Technical Report
131mpt-30bMosaicMLno page matchopenShots: 516.4%not datedStanford HELM
132Llama-2-7bMeta AIno page matchopenShots: 416.2%not datedBaichuan 2: Open Large-scale Language Models
133Llama-2-7bMeta AIno page matchopenShots: 816.0%not datedMixtral of Experts
134internlm/internlm-chat-20bEpoch: internlm-chat-20bopenShots: 015.7%not datedYi: Open Foundation Models by 01.AI
135LLaMA-13BMeta AIno page matchopenShots: 515.4%not datedStanford HELM
136mpt-30bMosaicMLno page matchopenShots: 815.2%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
137mpt-30bMosaicMLno page matchopenShots: 815.2%not datedQwen Technical Report
138Llama-2-7bMeta AIno page matchopendefault14.6%not datedGemma: Open Models Based on Gemini Research and Technology
139Llama-2-7bMeta AIno page matchopenShots: 814.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
140Llama-2-7bMeta AIno page matchopenShots: 014.6%not datedTextbooks Are All You Need II: phi-1.5 technical report
141meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 513.3%not datedStanford HELM
142LLaMA-7BMeta AIno page matchopenShots: 811.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
143LLaMA-7BMeta AIno page matchopendefault11.0%not datedLLaMA: Open and Efficient Foundation Language Models
144LLaMA-7BMeta AIno page matchopenShots: 811.0%not datedQwen Technical Report
145LLaMA-7BMeta AIno page matchopenShots: 011.0%not datedTextbooks Are All You Need II: phi-1.5 technical report
146LLaMA-7BMeta AIno page matchopenShots: 49.8%not datedBaichuan 2: Open Large-scale Language Models
147bigscience/bloomEpoch: bloom, Hugging Face,BigScienceopenShots: 59.5%not datedStanford HELM
148baichuan-inc/Baichuan-7BEpoch: Baichuan-7B, BaichuanopenShots: 49.2%not datedBaichuan 2: Open Large-scale Language Models
149mpt-7bMosaicMLno page matchopenShots: 89.1%not datedQwen Technical Report
150davinciOpenAIclosedShots: 59.0%not datedStanford HELM
151mpt-7bMosaicMLno page matchopenShots: 48.6%not datedBaichuan 2: Open Large-scale Language Models
152LLaMA-7BMeta AIno page matchopenShots: 58.0%not datedStanford HELM
153falcon-7bTechnology Innovation Instituteno page matchopenShots: 86.8%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
154falcon-7bTechnology Innovation Instituteno page matchopenShots: 86.8%not datedQwen Technical Report
155falcon-7bTechnology Innovation Instituteno page matchopenShots: 06.8%not datedTextbooks Are All You Need II: phi-1.5 technical report
156mpt-7bMosaicMLno page matchopenShots: 86.8%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
157mpt-7bMosaicMLno page matchopenShots: 06.8%not datedTextbooks Are All You Need II: phi-1.5 technical report
158falcon-7bTechnology Innovation Instituteno page matchopenShots: 45.5%not datedBaichuan 2: Open Large-scale Language Models
159falcon-7bTechnology Innovation Instituteno page matchopenShots: 54.6%not datedFalcon2-11B Technical Report
160deepseek-ai/deepseek-coder-1.3b-baseEpoch: deepseek-coder-1.3b-base, DeepSeek,Peking UniversityopenShots: 44.4%not datedQwen2.5-Coder Technical Report
161falcon-7bTechnology Innovation Instituteno page matchopenShots: 54.0%not datedStanford HELM
162opt-175bMeta AIno page matchopenShots: 54.0%not datedStanford HELM
163meta-llama/Llama-2-13b-chatEpoch: Llama-2-13b-chat, Meta AIopenShots: 42.7%not datedYi: Open Foundation Models by 01.AI
164facebook/opt-66bEpoch: opt-66b, Meta AIopenShots: 51.8%not datedStanford HELM
165curieOpenAInot statedShots: 51.6%not datedStanford HELM
166babbageOpenAInot statedShots: 50.7%not datedStanford HELM
167adaOpenAInot statedShots: 50.6%not datedStanford HELM
168text-curie-001OpenAIclosedShots: 50.6%not datedStanford HELM
169text-ada-001OpenAInot statedShots: 50.4%not datedStanford HELM
170text-babbage-001OpenAIclosedShots: 50.0%not datedStanford HELM

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method