Skip to content
Weights Not Walls
Menu

ANLI

Reported

Natural language inference (does one sentence follow from another) with examples collected to fool earlier models.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Link
http://arxiv.org/abs/2404.14219
Score shown
Epoch's column "Score", as a percentage, higher is better; random guessing scores about 33.3%. The source gives no error bars.
Results
15 in all: 8 for open-weight models, 7 for closed ones. 7 linked to a model page here.

Best open-weight result: microsoft/Phi-3-small-8k-instruct, 58.1%, number 2 of 15 results.

ANLI results
#ModelWeightsSettingScoreDateSource
1gpt-3.5-turbo-1106OpenAIcloseddefault58.1%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
2microsoft/Phi-3-small-8k-instructEpoch: Phi-3-small-8k-instruct, Microsoftopendefault58.1%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
3meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AIopendefault57.3%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
4microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, Microsoftopendefault55.8%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
5mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AIopendefault55.2%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
6microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, Microsoftopendefault52.8%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
7gemma-7bGoogle DeepMindno page matchopendefault48.7%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
8mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopendefault47.1%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
9microsoft/phi-2Epoch: phi-2, Microsoftopendefault42.5%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
10Megatron-Turing NLG 530BMicrosoft,NvidiaclosedShots: 139.7%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
11Megatron-Turing NLG 530BMicrosoft,NvidiaclosedShots: few39.6%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
12Megatron-Turing NLG 530BMicrosoft,NvidiaclosedShots: 036.6%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
13text-davinci-001OpenAIclosedShots: 035.4%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
14text-davinci-001OpenAIclosedShots: few34.0%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
15text-davinci-001OpenAIclosedShots: 133.9%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method