Skip to content
Weights Not Walls
Menu

LAMBADA

Reported

Predicting the last word of a passage, where the word can only be guessed from the wider context.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Link
https://arxiv.org/abs/2309.16609
Score shown
Epoch's column "Score", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
Results
53 in all: 37 for open-weight models, 16 for closed ones. 12 linked to a model page here.

Best open-weight result: falcon-180B, 79.8%, number 4 of 53 results.

LAMBADA results
#ModelWeightsSettingScoreDateSource
1Megatron-Turing NLG 530BMicrosoft,NvidiaclosedShots: few87.2%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
2text-davinci-001OpenAIclosedShots: few86.4%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
3text-davinci-001OpenAIclosedShots: few86.4%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
4falcon-180BTechnology Innovation Instituteno page matchopendefault79.8%not datedThe Falcon Series of Open Language Models
5meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopendefault78.9%not datedQwen Technical Report
6meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopendefault78.9%not datedQwen Technical Report
7Inflection-1Inflection AIcloseddefault78.5%not datedThe Falcon Series of Open Language Models
8PaLM 540BGoogle Researchcloseddefault77.9%not datedThe Falcon Series of Open Language Models
9LLaMA-65BMeta AIno page matchopendefault77.7%not datedQwen Technical Report
10LLaMA-65BMeta AIno page matchopendefault77.7%not datedQwen Technical Report
11Chinchilla (70B)DeepMindcloseddefault77.4%not datedThe Falcon Series of Open Language Models
12falcon-40bTechnology Innovation Instituteno page matchopendefault77.3%not datedThe Falcon Series of Open Language Models
13LLaMA-33BMeta AIno page matchopendefault77.2%not datedQwen Technical Report
14LLaMA-33BMeta AIno page matchopendefault77.2%not datedQwen Technical Report
15Megatron-Turing NLG 530BMicrosoft,Nvidiacloseddefault76.6%not datedThe Falcon Series of Open Language Models
16Megatron-Turing NLG 530BMicrosoft,NvidiaclosedShots: 076.6%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
17Llama-2-13bMeta AIno page matchopendefault76.5%not datedQwen Technical Report
18Llama-2-13bMeta AIno page matchopendefault76.5%not datedQwen Technical Report
19text-davinci-001OpenAIclosedShots: 076.2%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
20text-davinci-001OpenAIcloseddefault76.2%not datedThe Falcon Series of Open Language Models
21LLaMA-13BMeta AIno page matchopendefault75.2%not datedQwen Technical Report
22LLaMA-13BMeta AIno page matchopendefault75.2%not datedQwen Technical Report
23falcon-7bTechnology Innovation Instituteno page matchopendefault74.9%not datedThe Falcon Series of Open Language Models
24Gopher (280B)DeepMindclosedShots: 074.5%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
25Gopher (280B)DeepMindclosedShots: 074.5%not datedScaling Language Models: Methods, Analysis & Insights from Training Gopher
26Gopher (280B)DeepMindcloseddefault74.5%not datedThe Falcon Series of Open Language Models
27Baichuan-2-13B-BaseBaichuanno page matchopendefault74.0%not datedQwen Technical Report
28Baichuan-2-13B-BaseBaichuanno page matchopendefault74.0%not datedQwen Technical Report
29Baichuan-2-7B-BaseBaichuanno page matchopendefault73.3%not datedQwen Technical Report
30Baichuan-2-7B-BaseBaichuanno page matchopendefault73.3%not datedQwen Technical Report
31Llama-2-7bMeta AIno page matchopendefault73.3%not datedQwen Technical Report
32Llama-2-7bMeta AIno page matchopendefault73.3%not datedQwen Technical Report
33LLaMA-7BMeta AIno page matchopendefault73.3%not datedQwen Technical Report
34LLaMA-7BMeta AIno page matchopendefault73.3%not datedQwen Technical Report
35Megatron-Turing NLG 530BMicrosoft,NvidiaclosedShots: 173.1%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
36text-davinci-001OpenAIclosedShots: 172.5%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
37text-davinci-001OpenAIclosedShots: 172.5%not datedUsing DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
38internlm/internlm-20bEpoch: internlm-20bopendefault71.8%not datedQwen Technical Report
39internlm/internlm-20bEpoch: internlm-20bopendefault71.8%not datedQwen Technical Report
40stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AIopendefault71.3%not datedQwen Technical Report
41stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AIopendefault71.3%not datedQwen Technical Report
42Qwen-14BAlibabano page matchopendefault71.1%not datedQwen Technical Report
43Qwen-14BAlibabano page matchopendefault71.1%not datedQwen Technical Report
44mpt-7bMosaicMLno page matchopendefault70.0%not datedQwen Technical Report
45mpt-7bMosaicMLno page matchopendefault70.0%not datedQwen Technical Report
46Qwen-7BAlibabano page matchopendefault67.9%not datedQwen Technical Report
47Qwen-7BAlibabano page matchopendefault67.9%not datedQwen Technical Report
48internlm/internlm-7bEpoch: internlm-7bopendefault67.0%not datedQwen Technical Report
49internlm/internlm-7bEpoch: internlm-7bopendefault67.0%not datedQwen Technical Report
50Qwen/Qwen-1_8BEpoch: Qwen-1_8Bopendefault58.4%not datedQwen Technical Report
51Qwen/Qwen-1_8BEpoch: Qwen-1_8Bopendefault58.4%not datedQwen Technical Report
52zai-org/chatglm2-6bEpoch: chatglm2-6bopendefault54.3%not datedQwen Technical Report
53zai-org/chatglm2-6bEpoch: chatglm2-6bopendefault54.3%not datedQwen Technical Report

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method