LAMBADA
ReportedPredicting the last word of a passage, where the word can only be guessed from the wider context.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Link
- https://arxiv.org/abs/2309.16609
- Score shown
- Epoch's column "Score", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
- Results
- 53 in all: 37 for open-weight models, 16 for closed ones. 12 linked to a model page here.
Best open-weight result: falcon-180B, 79.8%, number 4 of 53 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | Megatron-Turing NLG 530BMicrosoft,Nvidia | closed | Shots: few | 87.2% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 2 | text-davinci-001OpenAI | closed | Shots: few | 86.4% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 3 | text-davinci-001OpenAI | closed | Shots: few | 86.4% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 4 | falcon-180BTechnology Innovation Instituteno page match | open | default | 79.8% | not dated | The Falcon Series of Open Language Models |
| 5 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | default | 78.9% | not dated | Qwen Technical Report |
| 6 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | default | 78.9% | not dated | Qwen Technical Report |
| 7 | Inflection-1Inflection AI | closed | default | 78.5% | not dated | The Falcon Series of Open Language Models |
| 8 | PaLM 540BGoogle Research | closed | default | 77.9% | not dated | The Falcon Series of Open Language Models |
| 9 | LLaMA-65BMeta AIno page match | open | default | 77.7% | not dated | Qwen Technical Report |
| 10 | LLaMA-65BMeta AIno page match | open | default | 77.7% | not dated | Qwen Technical Report |
| 11 | Chinchilla (70B)DeepMind | closed | default | 77.4% | not dated | The Falcon Series of Open Language Models |
| 12 | falcon-40bTechnology Innovation Instituteno page match | open | default | 77.3% | not dated | The Falcon Series of Open Language Models |
| 13 | LLaMA-33BMeta AIno page match | open | default | 77.2% | not dated | Qwen Technical Report |
| 14 | LLaMA-33BMeta AIno page match | open | default | 77.2% | not dated | Qwen Technical Report |
| 15 | Megatron-Turing NLG 530BMicrosoft,Nvidia | closed | default | 76.6% | not dated | The Falcon Series of Open Language Models |
| 16 | Megatron-Turing NLG 530BMicrosoft,Nvidia | closed | Shots: 0 | 76.6% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 17 | Llama-2-13bMeta AIno page match | open | default | 76.5% | not dated | Qwen Technical Report |
| 18 | Llama-2-13bMeta AIno page match | open | default | 76.5% | not dated | Qwen Technical Report |
| 19 | text-davinci-001OpenAI | closed | Shots: 0 | 76.2% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 20 | text-davinci-001OpenAI | closed | default | 76.2% | not dated | The Falcon Series of Open Language Models |
| 21 | LLaMA-13BMeta AIno page match | open | default | 75.2% | not dated | Qwen Technical Report |
| 22 | LLaMA-13BMeta AIno page match | open | default | 75.2% | not dated | Qwen Technical Report |
| 23 | falcon-7bTechnology Innovation Instituteno page match | open | default | 74.9% | not dated | The Falcon Series of Open Language Models |
| 24 | Gopher (280B)DeepMind | closed | Shots: 0 | 74.5% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 25 | Gopher (280B)DeepMind | closed | Shots: 0 | 74.5% | not dated | Scaling Language Models: Methods, Analysis & Insights from Training Gopher |
| 26 | Gopher (280B)DeepMind | closed | default | 74.5% | not dated | The Falcon Series of Open Language Models |
| 27 | Baichuan-2-13B-BaseBaichuanno page match | open | default | 74.0% | not dated | Qwen Technical Report |
| 28 | Baichuan-2-13B-BaseBaichuanno page match | open | default | 74.0% | not dated | Qwen Technical Report |
| 29 | Baichuan-2-7B-BaseBaichuanno page match | open | default | 73.3% | not dated | Qwen Technical Report |
| 30 | Baichuan-2-7B-BaseBaichuanno page match | open | default | 73.3% | not dated | Qwen Technical Report |
| 31 | Llama-2-7bMeta AIno page match | open | default | 73.3% | not dated | Qwen Technical Report |
| 32 | Llama-2-7bMeta AIno page match | open | default | 73.3% | not dated | Qwen Technical Report |
| 33 | LLaMA-7BMeta AIno page match | open | default | 73.3% | not dated | Qwen Technical Report |
| 34 | LLaMA-7BMeta AIno page match | open | default | 73.3% | not dated | Qwen Technical Report |
| 35 | Megatron-Turing NLG 530BMicrosoft,Nvidia | closed | Shots: 1 | 73.1% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 36 | text-davinci-001OpenAI | closed | Shots: 1 | 72.5% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 37 | text-davinci-001OpenAI | closed | Shots: 1 | 72.5% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 38 | internlm/internlm-20bEpoch: internlm-20b | open | default | 71.8% | not dated | Qwen Technical Report |
| 39 | internlm/internlm-20bEpoch: internlm-20b | open | default | 71.8% | not dated | Qwen Technical Report |
| 40 | stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AI | open | default | 71.3% | not dated | Qwen Technical Report |
| 41 | stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AI | open | default | 71.3% | not dated | Qwen Technical Report |
| 42 | Qwen-14BAlibabano page match | open | default | 71.1% | not dated | Qwen Technical Report |
| 43 | Qwen-14BAlibabano page match | open | default | 71.1% | not dated | Qwen Technical Report |
| 44 | mpt-7bMosaicMLno page match | open | default | 70.0% | not dated | Qwen Technical Report |
| 45 | mpt-7bMosaicMLno page match | open | default | 70.0% | not dated | Qwen Technical Report |
| 46 | Qwen-7BAlibabano page match | open | default | 67.9% | not dated | Qwen Technical Report |
| 47 | Qwen-7BAlibabano page match | open | default | 67.9% | not dated | Qwen Technical Report |
| 48 | internlm/internlm-7bEpoch: internlm-7b | open | default | 67.0% | not dated | Qwen Technical Report |
| 49 | internlm/internlm-7bEpoch: internlm-7b | open | default | 67.0% | not dated | Qwen Technical Report |
| 50 | Qwen/Qwen-1_8BEpoch: Qwen-1_8B | open | default | 58.4% | not dated | Qwen Technical Report |
| 51 | Qwen/Qwen-1_8BEpoch: Qwen-1_8B | open | default | 58.4% | not dated | Qwen Technical Report |
| 52 | zai-org/chatglm2-6bEpoch: chatglm2-6b | open | default | 54.3% | not dated | Qwen Technical Report |
| 53 | zai-org/chatglm2-6bEpoch: chatglm2-6b | open | default | 54.3% | not dated | Qwen Technical Report |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method