Skip to content
Weights Not Walls
Menu

Winogrande

Reported

Fill-in-the-blank sentences where the right answer needs common sense about who or what a word refers to.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Score shown
Epoch's column "Accuracy", as a percentage, higher is better; random guessing scores about 50.0%. The source gives no error bars.
Results
137 in all: 85 for open-weight models, 43 for closed ones, 9 where Epoch does not say. 36 linked to a model page here.

Best open-weight result: Llama-3.1-405B, 89.2%, number 1 of 137 results.

Winogrande results
#ModelWeightsSettingScoreDateSource
1Llama-3.1-405BMeta AIno page matchopenShots: 589.2%not datedDeepSeek-V3 Technical Report
2claude-3-opus-20240229AnthropicclosedShots: 588.5%not datedThe Claude 3 Model Family: Opus, Sonnet, Haiku
3gpt-4-0314OpenAIclosedShots: 587.5%not datedGPT-4 technical report
4gpt-4-32k-0314OpenAIclosedShots: 587.5%not datedThe Falcon Series of Open Language Models
5falcon-180BTechnology Innovation Instituteno page matchopenShots: 587.1%not datedThe Falcon Series of Open Language Models
6DeepSeek-V2DeepSeekno page matchopenShots: 586.3%not datedDeepSeek-V3 Technical Report
7deepseek-ai/DeepSeek-V3Epoch: DeepSeek-V3, DeepSeekopenShots: 585.2%not datedDeepSeek-V3 Technical Report
8PaLM 540BGoogle ResearchclosedShots: 585.1%not datedGPT-4 Technical report
9PaLM 540BGoogle ResearchclosedShots: few85.1%not datedPaLM: Scaling Language Modeling with Pathways
10deepseek-ai/DeepSeek-Coder-V2-BaseEpoch: DeepSeek-Coder-V2-Base, DeepSeekopendefault83.7%not datedQwen2.5-Coder Technical Report
11PaLM 540BGoogle ResearchclosedShots: 183.7%not datedPaLM: Scaling Language Modeling with Pathways
12Meta-Llama-3-70BMeta AIno page matchopenShots: 583.5%not datedThe Llama 3 Herd of Models
13PaLM 2-Lnot statedShots: 183.0%not datedPaLM 2 Technical Report
14Qwen2.5-72BAlibabano page matchopenShots: 582.3%not datedDeepSeek-V3 Technical Report
15meta-llama/Llama-3.1-405B-InstructEpoch: Llama-3.1-405B-Instruct, Meta AIopenShots: 582.2%not datedThe Llama 3 Herd of Models
16gpt-3.5-turbo-0613OpenAIclosedShots: 581.6%not datedThe Falcon Series of Open Language Models
17text-davinci-002OpenAIclosedShots: 581.6%not datedGPT-4 technical report
18microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, MicrosoftopenShots: 581.5%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
19microsoft/Phi-3-small-8k-instructEpoch: Phi-3-small-8k-instruct, MicrosoftopenShots: 581.5%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
20PaLM 540BGoogle ResearchclosedShots: 081.1%not datedPaLM: Scaling Language Modeling with Pathways
21PaLM 540BGoogle ResearchclosedShots: 081.1%not datedLLaMA: Open and Efficient Foundation Language Models
22Qwen2.5-Coder-32BAlibabano page matchopendefault80.8%not datedQwen2.5-Coder Technical Report
23meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopendefault80.2%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
24GLaM (MoE)GoogleclosedShots: 879.2%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
25PaLM 2-Mnot statedShots: 179.2%not datedPaLM 2 Technical Report
26gemma-7bGoogle DeepMindno page matchopendefault79.0%not datedGemma: Open Models Based on Gemini Research and Technology
27Megatron-Turing NLG 530BMicrosoft,Nvidiacloseddefault78.9%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
28tiiuae/falcon-11BEpoch: falcon-11b, Technology Innovation InstituteopenShots: 578.3%not datedFalcon2-11B Technical Report
29tiiuae/falcon-11BEpoch: falcon-11b, Technology Innovation InstituteopenShots: 578.3%not datedFalcon2-11B Technical Report
30Nemotron-4 15BNvidiaclosedShots: 078.0%not datedNemotron-4 15B Technical Report
31PaLM 2-Snot statedShots: 177.9%not datedPaLM 2 Technical Report
32text-davinci-001OpenAIclosedShots: few77.7%not datedLanguage Models are Few-Shot Learners
33text-davinci-001OpenAIclosedShots: 1677.7%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
34text-davinci-001OpenAIclosedShots: few77.7%not datedLanguage Models are Few-Shot Learners
35text-davinci-001OpenAIclosedShots: few77.7%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
36mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AIopendefault77.2%not datedMixtral of Experts
37LLaMA-65BMeta AIno page matchopenShots: 077.0%not datedLLaMA: Open and Efficient Foundation Language Models
38LLaMA-65BMeta AIno page matchopendefault77.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
39PaLM 62Bnot statedShots: 077.0%not datedPaLM: Scaling Language Modeling with Pathways
40PaLM 62Bnot statedShots: 077.0%not datedLLaMA: Open and Efficient Foundation Language Models
41falcon-40bTechnology Innovation Instituteno page matchopendefault76.9%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
42Qwen/Qwen2.5-Coder-14BEpoch: Qwen2.5-Coder-14Bopendefault76.8%not datedQwen2.5-Coder Technical Report
43Llama-2-34bMeta AIcloseddefault76.7%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
44Llama-2-34bMeta AIclosedShots: 076.7%not datedNemotron-4 15B Technical Report
45falcon-40bTechnology Innovation Instituteno page matchopenShots: 576.4%not datedFalcon2-11B Technical Report
46falcon-40bTechnology Innovation Instituteno page matchopenShots: 576.4%not datedFalcon2-11B Technical Report
47LLaMA-33BMeta AIno page matchopenShots: 076.0%not datedLLaMA: Open and Efficient Foundation Language Models
48LLaMA-33BMeta AIno page matchopendefault76.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
49Meta-Llama-3-8BMeta AIno page matchopenShots: 575.7%not datedThe Llama 3 Herd of Models
50mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 075.3%not datedTraining Compute-Optimal Large Language Models
51mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 075.3%not datedNemotron-4 15B Technical Report
52claude-3-sonnet-20240229AnthropicclosedShots: 575.1%not datedThe Claude 3 Model Family: Opus, Sonnet, Haiku
53Chinchilla (70B)DeepMindclosedShots: 074.9%not datedLLaMA: Open and Efficient Foundation Language Models
54Chinchilla (70B)DeepMindclosedShots: 074.9%not datedTraining Compute-Optimal Large Language Models
55claude-3-haiku-20240307AnthropicclosedShots: 574.2%not datedThe Claude 3 Model Family: Opus, Sonnet, Haiku
56mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopendefault74.2%not datedMixtral of Experts
57mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopendefault74.2%not datedGemma: Open Models Based on Gemini Research and Technology
58Megatron-Turing NLG 530BMicrosoft,NvidiaclosedShots: 173.7%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
59GLaM (MoE)GoogleclosedShots: 073.5%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
60microsoft/phi-1_5Epoch: phi-1_5, MicrosoftopenShots: 073.4%not datedTextbooks Are All You Need II: phi-1.5 technical report
61text-davinci-001OpenAIclosedShots: 173.2%not datedLanguage Models are Few-Shot Learners
62text-davinci-001OpenAIclosedShots: 173.2%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
63text-davinci-001OpenAIclosedShots: 173.2%not datedLanguage Models are Few-Shot Learners
64text-davinci-001OpenAIclosedShots: 173.2%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
65Megatron-Turing NLG 530BMicrosoft,NvidiaclosedShots: 073.0%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
66Megatron-Turing NLG 530BMicrosoft,NvidiaclosedShots: few73.0%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
67GLaM (MoE)GoogleclosedShots: 173.0%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
68LLaMA-13BMeta AIno page matchopenShots: 073.0%not datedLLaMA: Open and Efficient Foundation Language Models
69LLaMA-13BMeta AIno page matchopenShots: 073.0%not datedLLaMA: Open and Efficient Foundation Language Models
70LLaMA-13BMeta AIno page matchopendefault73.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
7101-ai/Yi-9BEpoch: Yi-9BopenShots: 573.0%not datedYi: Open Foundation Models by 01.AI
72deepseek-ai/DeepSeek-Coder-V2-Lite-BaseEpoch: DeepSeek-Coder-V2-Lite-Baseopendefault72.9%not datedQwen2.5-Coder Technical Report
73Qwen2.5-Coder-7BAlibabano page matchopendefault72.9%not datedQwen2.5-Coder Technical Report
74Llama-2-13bMeta AIno page matchopendefault72.8%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
75Llama-2-13bMeta AIno page matchopendefault72.8%not datedGemma: Open Models Based on Gemini Research and Technology
76Llama-2-13bMeta AIno page matchopenShots: 072.8%not datedNemotron-4 15B Technical Report
77gemma-7bGoogle DeepMindno page matchopenShots: 072.3%not datedNemotron-4 15B Technical Report
78Yi-6B01.AIno page matchopenShots: 571.3%not datedYi: Open Foundation Models by 01.AI
79mpt-30bMosaicMLno page matchopendefault71.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
80microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, MicrosoftopenShots: 570.8%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
81vicuna-13b-v1.1not statedShots: 070.8%not datedTextbooks Are All You Need II: phi-1.5 technical report
82Gopher (280B)DeepMindclosedShots: 070.2%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
83text-davinci-001OpenAIclosedShots: 070.2%not datedLLaMA: Open and Efficient Foundation Language Models
84text-davinci-001OpenAIclosedShots: 070.2%not datedLanguage Models are Few-Shot Learners
85text-davinci-001OpenAIclosedShots: 070.2%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
86text-davinci-001OpenAIclosedShots: 070.2%not datedLanguage Models are Few-Shot Learners
87text-davinci-001OpenAIclosedShots: 070.2%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
88Gopher (280B)DeepMindclosedShots: 070.1%not datedLLaMA: Open and Efficient Foundation Language Models
89Gopher (280B)DeepMindclosedShots: 070.1%not datedTraining Compute-Optimal Large Language Models
90Gopher (280B)DeepMindclosedShots: 070.1%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
91Gopher (280B)DeepMindclosedShots: 070.1%not datedScaling Language Models: Methods, Analysis & Insights from Training Gopher
92LLaMA-7BMeta AIno page matchopenShots: 070.1%not datedLLaMA: Open and Efficient Foundation Language Models
93LLaMA-7BMeta AIno page matchopenShots: 070.1%not datedLLaMA: Open and Efficient Foundation Language Models
94LLaMA-7BMeta AIno page matchopendefault70.1%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
95LLaMA-7BMeta AIno page matchopenShots: 069.6%not datedXGen-7B Technical Report
96Llama-2-7bMeta AIno page matchopendefault69.2%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
97Llama-2-7bMeta AIno page matchopendefault69.2%not datedGemma: Open Models Based on Gemini Research and Technology
98Llama-2-7bMeta AIno page matchopenShots: 069.1%not datedTextbooks Are All You Need II: phi-1.5 technical report
99gpt-3.5-turbo-1106OpenAIclosedShots: 568.8%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
100mpt-7bMosaicMLno page matchopenShots: 068.6%not datedXGen-7B Technical Report
101mpt-7bMosaicMLno page matchopendefault68.3%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
102mpt-7bMosaicMLno page matchopenShots: 068.0%not datedTextbooks Are All You Need II: phi-1.5 technical report
103Qwen/Qwen2.5-Coder-3BEpoch: Qwen2.5-Coder-3Bopendefault67.4%not datedQwen2.5-Coder Technical Report
104falcon-7bTechnology Innovation Instituteno page matchopenShots: 067.2%not datedXGen-7B Technical Report
105falcon-7bTechnology Innovation Instituteno page matchopenShots: 567.2%not datedFalcon2-11B Technical Report
106falcon-7bTechnology Innovation Instituteno page matchopenShots: 567.2%not datedFalcon2-11B Technical Report
107open_llama_7bnot statedShots: 067.0%not datedXGen-7B Technical Report
108LLaMA-7BMeta AIno page matchopenShots: 066.9%not datedTextbooks Are All You Need II: phi-1.5 technical report
109falcon-7bTechnology Innovation Instituteno page matchopendefault66.3%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
110falcon-7bTechnology Innovation Instituteno page matchopenShots: 066.2%not datedTextbooks Are All You Need II: phi-1.5 technical report
111EleutherAI/gpt-neox-20bEpoch: gpt-neox-20b, EleutherAIopenShots: 066.1%not datedXGen-7B Technical Report
112INTELLECT-1-InstructPrime Intellect,Hugging Face,Arcee AInot statedShots: 565.8%not datedINTELLECT-1 Technical Report
113gemma-2bGoogle DeepMindno page matchopendefault65.4%not datedGemma: Open Models Based on Gemini Research and Technology
114meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AIopenShots: 565.0%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
115Salesforce/xgen-7b-8k-baseEpoch: xgen-7b-8k-base, SalesforceopenShots: 064.9%not datedXGen-7B Technical Report
116facebook/opt-13bEpoch: opt-13bopenShots: 064.7%not datedXGen-7B Technical Report
117gpt-j-6bEleutherAI,LAIONno page matchopenShots: 064.5%not datedXGen-7B Technical Report
118bigcode/starcoder2-15bEpoch: starcoder2-15b, Hugging Face,ServiceNow,Nvidia,BigCodeopendefault64.3%not datedQwen2.5-Coder Technical Report
119RedPajama-INCITE-7B-Basenot statedShots: 063.8%not datedXGen-7B Technical Report
120deepseek-ai/deepseek-coder-33b-baseEpoch: deepseek-coder-33b-base, DeepSeek,Peking Universityopendefault62.0%not datedQwen2.5-Coder Technical Report
121mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AIopenShots: 562.0%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
122dolly-v2-12bDatabricksno page matchopenShots: 061.8%not datedXGen-7B Technical Report
123facebook/opt-1.3bEpoch: opt-1.3b, Meta AIopenShots: 061.0%not datedTextbooks Are All You Need II: phi-1.5 technical report
124Cerebras-GPT-13BCerebras Systemsno page matchopenShots: 060.8%not datedXGen-7B Technical Report
125Qwen2.5-Coder-1.5BAlibabano page matchopendefault60.7%not datedQwen2.5-Coder Technical Report
126Qwen/CodeQwen1.5-7BEpoch: CodeQwen1.5-7Bopendefault59.8%not datedQwen2.5-Coder Technical Report
127gpt2-xlOpenAIno page matchopendefault58.3%not datedvia Epoch AI
128gpt2-xlOpenAIno page matchopenShots: 058.3%not datedTextbooks Are All You Need II: phi-1.5 technical report
129deepseek-ai/deepseek-coder-6.7b-baseEpoch: deepseek-coder-6.7b-base, DeepSeek,Peking Universityopendefault57.6%not datedQwen2.5-Coder Technical Report
130bigcode/starcoder2-3bEpoch: starcoder2-3b, Hugging Face,ServiceNow,Nvidia,BigCodeopendefault57.1%not datedQwen2.5-Coder Technical Report
131bigcode/starcoder2-7bEpoch: starcoder2-7b, Hugging Face,ServiceNow,Nvidia,BigCodeopendefault57.1%not datedQwen2.5-Coder Technical Report
132gemma-7bGoogle DeepMindno page matchopenShots: 555.6%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
133Qwen/Qwen2.5-Coder-0.5BEpoch: Qwen2.5-Coder-0.5Bopendefault54.8%not datedQwen2.5-Coder Technical Report
134microsoft/phi-2Epoch: phi-2, MicrosoftopenShots: 554.7%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
135mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 554.2%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
136deepseek-ai/deepseek-coder-1.3b-baseEpoch: deepseek-coder-1.3b-base, DeepSeek,Peking Universityopendefault53.3%not datedQwen2.5-Coder Technical Report
137stabilityai/stablelm-tuned-alpha-7bEpoch: stablelm-tuned-alpha-7bopenShots: 051.5%not datedXGen-7B Technical Report

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method