Skip to content
Weights Not Walls
Menu

OpenBookQA

Reported

Elementary science questions that need a given fact plus common knowledge to answer.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Score shown
Epoch's column "Accuracy", as a percentage, higher is better; random guessing scores about 25.0%. The source gives no error bars.
Results
70 in all: 47 for open-weight models, 16 for closed ones, 7 where Epoch does not say. 19 linked to a model page here.

Best open-weight result: microsoft/Phi-3-mini-4k-instruct, 88.0%, number 1 of 70 results.

OpenBookQA results
#ModelWeightsSettingScoreDateSource
1microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, MicrosoftopenShots: 1088.0%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
2microsoft/Phi-3-small-8k-instructEpoch: Phi-3-small-8k-instruct, MicrosoftopenShots: 1088.0%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
3microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, MicrosoftopenShots: 1087.4%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
4microsoft/Phi-3-medium-128k-instructEpoch: Phi-3-medium-128k-instruct, MicrosoftopenShots: 1087.4%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
5gpt-3.5-turbo-1106OpenAIclosedShots: 1086.0%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
6mistralai/Mixtral-8x7B-v0.1Epoch: Mixtral-8x7B-v0.1, Mistral AIopenShots: 1085.8%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
7microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, MicrosoftopenShots: 1083.2%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
8microsoft/Phi-3-mini-4k-instructEpoch: Phi-3-mini-4k-instruct, MicrosoftopenShots: 1083.2%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
9meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AIopenShots: 1082.6%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
10mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AIopenShots: 1079.8%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
11gemma-7bGoogle DeepMindno page matchopenShots: 1078.6%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
12microsoft/phi-2Epoch: phi-2, MicrosoftopenShots: 1073.6%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
13microsoft/phi-2Epoch: phi-2, MicrosoftopenShots: 1073.6%not datedPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
14PaLM 540BGoogle ResearchclosedShots: few68.0%not datedPaLM: Scaling Language Modeling with Pathways
15text-davinci-001OpenAIclosedShots: 10065.4%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
16text-davinci-001OpenAIclosedShots: few65.4%not datedLanguage Models are Few-Shot Learners
17falcon-180BTechnology Innovation Instituteno page matchopenShots: 164.2%not datedThe Falcon Series of Open Language Models
18GLaM (MoE)GoogleclosedShots: 3263.0%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
19meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AIopenShots: 060.2%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
20LLaMA-65BMeta AIno page matchopenShots: 060.2%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
21LLaMA-65BMeta AIno page matchopenShots: 060.2%not datedLLaMA: Open and Efficient Foundation Language Models
22text-davinci-001OpenAIclosedShots: 158.8%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
23text-davinci-001OpenAIclosedShots: 158.8%not datedLanguage Models are Few-Shot Learners
24Llama-2-7bMeta AIno page matchopenShots: 058.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
25LLaMA-33BMeta AIno page matchopenShots: 058.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
26LLaMA-33BMeta AIno page matchopenShots: 058.6%not datedLLaMA: Open and Efficient Foundation Language Models
27Llama-2-34bMeta AIclosedShots: 058.2%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
28text-davinci-001OpenAIclosedShots: 057.6%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
29text-davinci-001OpenAIclosedShots: 057.6%not datedLanguage Models are Few-Shot Learners
30text-davinci-001OpenAIclosedShots: 057.6%not datedLLaMA: Open and Efficient Foundation Language Models
31PaLM 2-Mnot statedShots: 157.4%not datedPaLM 2 Technical Report
32LLaMA-7BMeta AIno page matchopenShots: 057.2%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
33LLaMA-7BMeta AIno page matchopenShots: 057.2%not datedLLaMA: Open and Efficient Foundation Language Models
34Llama-2-13bMeta AIno page matchopenShots: 057.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
35falcon-40bTechnology Innovation Instituteno page matchopenShots: 056.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
36LLaMA-13BMeta AIno page matchopenShots: 056.4%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
37LLaMA-13BMeta AIno page matchopenShots: 056.4%not datedLLaMA: Open and Efficient Foundation Language Models
38PaLM 2-Snot statedShots: 156.2%not datedPaLM 2 Technical Report
39GLaM (MoE)GoogleclosedShots: 155.2%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
40PaLM 2-Snot statedShots: 153.6%not datedPaLM 2 Technical Report
41PaLM 540BGoogle ResearchclosedShots: 153.6%not datedPaLM: Scaling Language Modeling with Pathways
42GLaM (MoE)GoogleclosedShots: 053.4%not datedGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
43PaLM 540BGoogle ResearchclosedShots: 053.4%not datedPaLM: Scaling Language Modeling with Pathways
44PaLM 540BGoogle ResearchclosedShots: 053.4%not datedLLaMA: Open and Efficient Foundation Language Models
45mpt-30bMosaicMLno page matchopenShots: 052.0%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
46falcon-7bTechnology Innovation Instituteno page matchopenShots: 051.6%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
47mpt-7bMosaicMLno page matchopenShots: 051.4%not datedLlama 2: Open Foundation and Fine-Tuned Chat Models
48PaLM 62Bnot statedShots: 050.4%not datedLLaMA: Open and Efficient Foundation Language Models
49meta-llama/Meta-Llama-3-8B-InstructEpoch: Meta-Llama-3-8B-Instruct, Meta AIopenShots: 045.0%not datedThe Llama 3 Herd of Models
50LLaMA-7BMeta AIno page matchopenShots: 044.2%not datedXGen-7B Technical Report
51falcon-7bTechnology Innovation Instituteno page matchopenShots: 044.0%not datedXGen-7B Technical Report
52mpt-7bMosaicMLno page matchopenShots: 041.8%not datedXGen-7B Technical Report
53Salesforce/xgen-7b-8k-baseEpoch: xgen-7b-8k-base, SalesforceopenShots: 040.2%not datedXGen-7B Technical Report
54RedPajama-INCITE-7B-Basenot statedShots: 040.0%not datedXGen-7B Technical Report
55facebook/opt-13bEpoch: opt-13bopenShots: 039.8%not datedXGen-7B Technical Report
56dolly-v2-12bDatabricksno page matchopenShots: 039.2%not datedXGen-7B Technical Report
57open_llama_7bnot statedShots: 039.0%not datedXGen-7B Technical Report
58EleutherAI/gpt-neox-20bEpoch: gpt-neox-20b, EleutherAIopenShots: 038.8%not datedXGen-7B Technical Report
59gpt-j-6bEleutherAI,LAIONno page matchopenShots: 038.2%not datedXGen-7B Technical Report
60microsoft/phi-1_5Epoch: phi-1_5, MicrosoftopenShots: 037.2%not datedTextbooks Are All You Need II: phi-1.5 technical report
61Cerebras-GPT-13BCerebras Systemsno page matchopenShots: 035.8%not datedXGen-7B Technical Report
62vicuna-13b-v1.1not statedShots: 033.0%not datedTextbooks Are All You Need II: phi-1.5 technical report
63stabilityai/stablelm-tuned-alpha-7bEpoch: stablelm-tuned-alpha-7bopenShots: 032.4%not datedXGen-7B Technical Report
64falcon-7bTechnology Innovation Instituteno page matchopenShots: 032.0%not datedTextbooks Are All You Need II: phi-1.5 technical report
65Llama-2-7bMeta AIno page matchopenShots: 031.4%not datedTextbooks Are All You Need II: phi-1.5 technical report
66mpt-7bMosaicMLno page matchopenShots: 031.4%not datedTextbooks Are All You Need II: phi-1.5 technical report
67LLaMA-7BMeta AIno page matchopenShots: 028.4%not datedTextbooks Are All You Need II: phi-1.5 technical report
68facebook/opt-1.3bEpoch: opt-1.3b, Meta AIopenShots: 024.0%not datedTextbooks Are All You Need II: phi-1.5 technical report
69gpt-neo-2.7BEleutherAIno page matchopenShots: 023.2%not datedTextbooks Are All You Need II: phi-1.5 technical report
70gpt2-xlOpenAIno page matchopenShots: 022.4%not datedTextbooks Are All You Need II: phi-1.5 technical report

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method