OpenBookQA
ReportedElementary science questions that need a given fact plus common knowledge to answer.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Score shown
- Epoch's column "Accuracy", as a percentage, higher is better; random guessing scores about 25.0%. The source gives no error bars.
- Results
- 70 in all: 47 for open-weight models, 16 for closed ones, 7 where Epoch does not say. 19 linked to a model page here.
Best open-weight result: microsoft/Phi-3-mini-4k-instruct, 88.0%, number 1 of 70 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, Microsoft | open | Shots: 10 | 88.0% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 2 | microsoft/Phi-3-small-8k-instructEpoch: Phi-3-small-8k-instruct, Microsoft | open | Shots: 10 | 88.0% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 3 | microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, Microsoft | open | Shots: 10 | 87.4% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 4 | microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, Microsoft | open | Shots: 10 | 87.4% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 5 | gpt-3.5-turbo-1106OpenAI | closed | Shots: 10 | 86.0% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 6 | mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AI | open | Shots: 10 | 85.8% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 7 | microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, Microsoft | open | Shots: 10 | 83.2% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 8 | microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, Microsoft | open | Shots: 10 | 83.2% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 9 | meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AI | open | Shots: 10 | 82.6% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 10 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 10 | 79.8% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 11 | gemma-7bGoogle DeepMindno page match | open | Shots: 10 | 78.6% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 12 | microsoft/phi-2Epoch: phi-2, Microsoft | open | Shots: 10 | 73.6% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 13 | microsoft/phi-2Epoch: phi-2, Microsoft | open | Shots: 10 | 73.6% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 14 | PaLM 540BGoogle Research | closed | Shots: few | 68.0% | not dated | PaLM: Scaling Language Modeling with Pathways |
| 15 | text-davinci-001OpenAI | closed | Shots: 100 | 65.4% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 16 | text-davinci-001OpenAI | closed | Shots: few | 65.4% | not dated | Language Models are Few-Shot Learners |
| 17 | falcon-180BTechnology Innovation Instituteno page match | open | Shots: 1 | 64.2% | not dated | The Falcon Series of Open Language Models |
| 18 | GLaM (MoE)Google | closed | Shots: 32 | 63.0% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 19 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 0 | 60.2% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 20 | LLaMA-65BMeta AIno page match | open | Shots: 0 | 60.2% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 21 | LLaMA-65BMeta AIno page match | open | Shots: 0 | 60.2% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 22 | text-davinci-001OpenAI | closed | Shots: 1 | 58.8% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 23 | text-davinci-001OpenAI | closed | Shots: 1 | 58.8% | not dated | Language Models are Few-Shot Learners |
| 24 | Llama-2-7bMeta AIno page match | open | Shots: 0 | 58.6% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 25 | LLaMA-33BMeta AIno page match | open | Shots: 0 | 58.6% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 26 | LLaMA-33BMeta AIno page match | open | Shots: 0 | 58.6% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 27 | Llama-2-34bMeta AI | closed | Shots: 0 | 58.2% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 28 | text-davinci-001OpenAI | closed | Shots: 0 | 57.6% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 29 | text-davinci-001OpenAI | closed | Shots: 0 | 57.6% | not dated | Language Models are Few-Shot Learners |
| 30 | text-davinci-001OpenAI | closed | Shots: 0 | 57.6% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 31 | PaLM 2-M | not stated | Shots: 1 | 57.4% | not dated | PaLM 2 Technical Report |
| 32 | LLaMA-7BMeta AIno page match | open | Shots: 0 | 57.2% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 33 | LLaMA-7BMeta AIno page match | open | Shots: 0 | 57.2% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 34 | Llama-2-13bMeta AIno page match | open | Shots: 0 | 57.0% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 35 | falcon-40bTechnology Innovation Instituteno page match | open | Shots: 0 | 56.6% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 36 | LLaMA-13BMeta AIno page match | open | Shots: 0 | 56.4% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 37 | LLaMA-13BMeta AIno page match | open | Shots: 0 | 56.4% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 38 | PaLM 2-S | not stated | Shots: 1 | 56.2% | not dated | PaLM 2 Technical Report |
| 39 | GLaM (MoE)Google | closed | Shots: 1 | 55.2% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 40 | PaLM 2-S | not stated | Shots: 1 | 53.6% | not dated | PaLM 2 Technical Report |
| 41 | PaLM 540BGoogle Research | closed | Shots: 1 | 53.6% | not dated | PaLM: Scaling Language Modeling with Pathways |
| 42 | GLaM (MoE)Google | closed | Shots: 0 | 53.4% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 43 | PaLM 540BGoogle Research | closed | Shots: 0 | 53.4% | not dated | PaLM: Scaling Language Modeling with Pathways |
| 44 | PaLM 540BGoogle Research | closed | Shots: 0 | 53.4% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 45 | mpt-30bMosaicMLno page match | open | Shots: 0 | 52.0% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 46 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 0 | 51.6% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 47 | mpt-7bMosaicMLno page match | open | Shots: 0 | 51.4% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 48 | PaLM 62B | not stated | Shots: 0 | 50.4% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 49 | meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AI | open | Shots: 0 | 45.0% | not dated | The Llama 3 Herd of Models |
| 50 | LLaMA-7BMeta AIno page match | open | Shots: 0 | 44.2% | not dated | XGen-7B Technical Report |
| 51 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 0 | 44.0% | not dated | XGen-7B Technical Report |
| 52 | mpt-7bMosaicMLno page match | open | Shots: 0 | 41.8% | not dated | XGen-7B Technical Report |
| 53 | Salesforce/xgen-7b-8k-baseEpoch: xgen-7b-8k-base, Salesforce | open | Shots: 0 | 40.2% | not dated | XGen-7B Technical Report |
| 54 | RedPajama-INCITE-7B-Base | not stated | Shots: 0 | 40.0% | not dated | XGen-7B Technical Report |
| 55 | facebook/opt-13bEpoch: opt-13b | open | Shots: 0 | 39.8% | not dated | XGen-7B Technical Report |
| 56 | dolly-v2-12bDatabricksno page match | open | Shots: 0 | 39.2% | not dated | XGen-7B Technical Report |
| 57 | open_llama_7b | not stated | Shots: 0 | 39.0% | not dated | XGen-7B Technical Report |
| 58 | EleutherAI/gpt-neox-20bEpoch: gpt-neox-20b, EleutherAI | open | Shots: 0 | 38.8% | not dated | XGen-7B Technical Report |
| 59 | gpt-j-6bEleutherAI,LAIONno page match | open | Shots: 0 | 38.2% | not dated | XGen-7B Technical Report |
| 60 | microsoft/phi-1_5Epoch: phi-1_5, Microsoft | open | Shots: 0 | 37.2% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 61 | Cerebras-GPT-13BCerebras Systemsno page match | open | Shots: 0 | 35.8% | not dated | XGen-7B Technical Report |
| 62 | vicuna-13b-v1.1 | not stated | Shots: 0 | 33.0% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 63 | stabilityai/stablelm-tuned-alpha-7bEpoch: stablelm-tuned-alpha-7b | open | Shots: 0 | 32.4% | not dated | XGen-7B Technical Report |
| 64 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 0 | 32.0% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 65 | Llama-2-7bMeta AIno page match | open | Shots: 0 | 31.4% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 66 | mpt-7bMosaicMLno page match | open | Shots: 0 | 31.4% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 67 | LLaMA-7BMeta AIno page match | open | Shots: 0 | 28.4% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 68 | facebook/opt-1.3bEpoch: opt-1.3b, Meta AI | open | Shots: 0 | 24.0% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 69 | gpt-neo-2.7BEleutherAIno page match | open | Shots: 0 | 23.2% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 70 | gpt2-xlOpenAIno page match | open | Shots: 0 | 22.4% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method