BoolQ
ReportedYes or no questions about a short passage.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Score shown
- Epoch's column "Score", as a percentage, higher is better. The source gives no error bars.
- Results
- 136 in all: 88 for open-weight models, 33 for closed ones, 15 where Epoch does not say. 24 linked to a model page here.
Best open-weight result: T5-11B, 91.2%, number 1 of 136 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | T5-11BGoogleno page match | open | default | 91.2% | not dated | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer |
| 2 | PaLM 2-L | not stated | Shots: 1 | 90.9% | not dated | The Falcon Series of Open Language Models |
| 3 | T5-3BGoogleno page match | open | default | 89.9% | not dated | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer |
| 4 | Inflection-1Inflection AI | closed | default | 89.7% | not dated | The Falcon Series of Open Language Models |
| 5 | stabilityai/StableBeluga2Epoch: StableBeluga2, Stability AI | open | default | 89.4% | not dated | Qwen Technical Report |
| 6 | falcon-180BTechnology Innovation Instituteno page match | open | Shots: 1 | 89.0% | not dated | The Falcon Series of Open Language Models |
| 7 | gpt-4o-mini-2024-07-18OpenAI | closed | Shots: 2 | 88.7% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 8 | PaLM 540BGoogle Research | closed | Shots: 1 | 88.7% | not dated | The Falcon Series of Open Language Models |
| 9 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | Shots: 5 | 88.6% | not dated | Stanford HELM |
| 10 | PaLM 2-M | not stated | Shots: 1 | 88.6% | not dated | The Falcon Series of Open Language Models |
| 11 | PaLM 2-S | not stated | Shots: 1 | 88.1% | not dated | The Falcon Series of Open Language Models |
| 12 | text-davinci-003OpenAI | not stated | Shots: 5 | 88.1% | not dated | Stanford HELM |
| 13 | PaLM 540BGoogle Research | closed | Shots: 0 | 88.0% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 14 | PaLM 540BGoogle Research | closed | default | 88.0% | not dated | The Falcon Series of Open Language Models |
| 15 | falcon-180BTechnology Innovation Instituteno page match | open | default | 87.8% | not dated | The Falcon Series of Open Language Models |
| 16 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | default | 87.7% | not dated | Qwen Technical Report |
| 17 | text-davinci-002OpenAI | closed | Shots: 5 | 87.7% | not dated | Stanford HELM |
| 18 | internlm/internlm-20bEpoch: internlm-20b | open | default | 87.5% | not dated | Qwen Technical Report |
| 19 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 5 | 87.4% | not dated | Stanford HELM |
| 20 | LLaMA-65BMeta AIno page match | open | Shots: 5 | 87.1% | not dated | Stanford HELM |
| 21 | gpt-3.5-turbo-0613OpenAI | closed | Shots: 5 | 87.0% | not dated | Stanford HELM |
| 22 | LLaMA-65BMeta AIno page match | open | default | 86.6% | not dated | Qwen Technical Report |
| 23 | Qwen-14BAlibabano page match | open | default | 86.2% | not dated | Qwen Technical Report |
| 24 | LLaMA-33BMeta AIno page match | open | Shots: 5 | 86.1% | not dated | Stanford HELM |
| 25 | gemini-1.5-flash-001Google DeepMind | closed | Shots: 2 | 85.8% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 26 | gemma-2-9bGoogle DeepMindno page match | open | Shots: 2 | 85.7% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 27 | T5-Large | not stated | default | 85.4% | not dated | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer |
| 28 | LLaMA-65BMeta AIno page match | open | default | 85.3% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 29 | LLaMA-65BMeta AIno page match | open | Shots: 0 | 85.3% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 30 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | default | 85.0% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 31 | meta-llama/Llama-2-70b-hfEpoch: Llama-2-70b-hf, Meta AI | open | default | 85.0% | not dated | The Falcon Series of Open Language Models |
| 32 | mpt-30b-instructMosaicMLno page match | open | Shots: 5 | 85.0% | not dated | Stanford HELM |
| 33 | Megatron-Turing NLG 530BMicrosoft,Nvidia | closed | Shots: few | 84.8% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 34 | PaLM 62B | not stated | Shots: 0 | 84.8% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 35 | microsoft/Phi-3.5-MoE-instructEpoch: Phi-3.5-MoE-instruct, Microsoft | open | Shots: 2 | 84.6% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 36 | LLaMA-33BMeta AIno page match | open | default | 84.4% | not dated | Qwen Technical Report |
| 37 | Chinchilla (70B)DeepMind | closed | Shots: 0 | 83.7% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 38 | Chinchilla (70B)DeepMind | closed | default | 83.7% | not dated | The Falcon Series of Open Language Models |
| 39 | Chinchilla (70B)DeepMind | closed | Shots: 0 | 83.7% | not dated | Training Compute-Optimal Large Language Models |
| 40 | Llama-2-34bMeta AI | closed | default | 83.7% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 41 | Llama-2-34bMeta AI | closed | default | 83.7% | not dated | The Falcon Series of Open Language Models |
| 42 | vicuna-13b-v1.1 | not stated | Shots: 0 | 83.5% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 43 | gemma-7bGoogle DeepMindno page match | open | Shots: 0 | 83.2% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 44 | mistralai/Mistral-7B-Instruct-v0.2Epoch: Mistral-7B-Instruct-v0.2, Mistral AI | open | Shots: 0 | 83.2% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 45 | falcon-40bTechnology Innovation Instituteno page match | open | default | 83.1% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 46 | LLaMA-33BMeta AIno page match | open | default | 83.1% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 47 | LLaMA-33BMeta AIno page match | open | Shots: 0 | 83.1% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 48 | meta-llama/Llama-3.1-8B-InstructEpoch: Llama-3.1-8B-Instruct, Meta AI | open | Shots: 2 | 82.8% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 49 | Megatron-Turing NLG 530BMicrosoft,Nvidia | closed | Shots: 1 | 82.5% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 50 | mistralai/Mistral-Nemo-Base-2407Epoch: Mistral-Nemo-Base-2407, Mistral AI | open | Shots: 2 | 82.5% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 51 | Llama-2-13bMeta AIno page match | open | default | 82.4% | not dated | Qwen Technical Report |
| 52 | falcon-40bTechnology Innovation Instituteno page match | open | Shots: 5 | 81.9% | not dated | Stanford HELM |
| 53 | falcon-40bTechnology Innovation Instituteno page match | open | default | 81.9% | not dated | The Falcon Series of Open Language Models |
| 54 | Llama-2-13bMeta AIno page match | open | Shots: 0 | 81.7% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 55 | Llama-2-13bMeta AIno page match | open | default | 81.7% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 56 | Llama-2-13bMeta AIno page match | open | default | 81.7% | not dated | The Falcon Series of Open Language Models |
| 57 | T5-Base | not stated | default | 81.4% | not dated | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer |
| 58 | Llama-2-13bMeta AIno page match | open | Shots: 5 | 81.1% | not dated | Stanford HELM |
| 59 | vicuna-13b-v1.3Large Model Systems Organization,University of California (UC) Berkeleyno page match | open | Shots: 5 | 80.8% | not dated | Stanford HELM |
| 60 | mistralai/Mistral-7B-v0.1Epoch: Mistral-7B-v0.1, Mistral AI | open | Shots: 2 | 80.5% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 61 | Gopher (280B)DeepMind | closed | default | 79.4% | not dated | The Falcon Series of Open Language Models |
| 62 | Gopher (280B)DeepMind | closed | Shots: 0 | 79.3% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 63 | Gopher (280B)DeepMind | closed | Shots: 0 | 79.3% | not dated | Training Compute-Optimal Large Language Models |
| 64 | opt-175bMeta AIno page match | open | Shots: 5 | 79.3% | not dated | Stanford HELM |
| 65 | zai-org/chatglm2-6bEpoch: chatglm2-6b | open | default | 79.0% | not dated | Qwen Technical Report |
| 66 | mpt-30bMosaicMLno page match | open | default | 79.0% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 67 | LLaMA-13BMeta AIno page match | open | default | 78.7% | not dated | Qwen Technical Report |
| 68 | Megatron-Turing NLG 530BMicrosoft,Nvidia | closed | Shots: 0 | 78.2% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 69 | Megatron-Turing NLG 530BMicrosoft,Nvidia | closed | default | 78.2% | not dated | The Falcon Series of Open Language Models |
| 70 | Megatron-Turing NLG 530BMicrosoft,Nvidia | closed | Shots: 0 | 78.2% | not dated | Training Compute-Optimal Large Language Models |
| 71 | LLaMA-13BMeta AIno page match | open | default | 78.1% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 72 | LLaMA-13BMeta AIno page match | open | Shots: 0 | 78.1% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 73 | microsoft/Phi-3.5-mini-instructEpoch: Phi-3.5-mini-instruct, Microsoft | open | Shots: 2 | 78.0% | not dated | Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone |
| 74 | Llama-2-7bMeta AIno page match | open | Shots: 0 | 77.9% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 75 | text-davinci-001OpenAI | closed | Shots: few | 77.5% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 76 | text-davinci-001OpenAI | closed | Shots: few | 77.5% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 77 | Llama-2-7bMeta AIno page match | open | default | 77.4% | not dated | Qwen Technical Report |
| 78 | Llama-2-7bMeta AIno page match | open | Shots: 0 | 77.4% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 79 | Llama-2-7bMeta AIno page match | open | default | 77.4% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 80 | Llama-2-7bMeta AIno page match | open | default | 77.4% | not dated | The Falcon Series of Open Language Models |
| 81 | text-davinci-001OpenAI | closed | Shots: 1 | 76.7% | not dated | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts |
| 82 | text-davinci-001OpenAI | closed | Shots: 1 | 76.7% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 83 | LLaMA-7BMeta AIno page match | open | default | 76.5% | not dated | Qwen Technical Report |
| 84 | LLaMA-7BMeta AIno page match | open | default | 76.5% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 85 | LLaMA-7BMeta AIno page match | open | Shots: 0 | 76.5% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 86 | Qwen-7BAlibabano page match | open | default | 76.4% | not dated | Qwen Technical Report |
| 87 | T5-Small | not stated | default | 76.4% | not dated | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer |
| 88 | text-davinci-001OpenAI | closed | Shots: few | 76.4% | not dated | Language Models are Few-Shot Learners |
| 89 | Llama-2-7bMeta AIno page match | open | Shots: 5 | 76.2% | not dated | Stanford HELM |
| 90 | T5-11BGoogleno page match | open | default | 76.1% | not dated | Stanford HELM |
| 91 | facebook/opt-66bEpoch: opt-66b, Meta AI | open | Shots: 5 | 76.0% | not dated | Stanford HELM |
| 92 | microsoft/phi-1_5Epoch: phi-1_5, Microsoft | open | Shots: 0 | 75.8% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 93 | LLaMA-7BMeta AIno page match | open | Shots: 5 | 75.6% | not dated | Stanford HELM |
| 94 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 5 | 75.3% | not dated | Stanford HELM |
| 95 | mpt-7bMosaicMLno page match | open | default | 75.0% | not dated | Qwen Technical Report |
| 96 | mpt-7bMosaicMLno page match | open | default | 75.0% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 97 | LLaMA-7BMeta AIno page match | open | Shots: 0 | 74.9% | not dated | XGen-7B Technical Report |
| 98 | Salesforce/xgen-7b-8k-baseEpoch: xgen-7b-8k-base, Salesforce | open | Shots: 0 | 74.3% | not dated | XGen-7B Technical Report |
| 99 | mpt-7bMosaicMLno page match | open | Shots: 0 | 74.1% | not dated | XGen-7B Technical Report |
| 100 | mpt-7bMosaicMLno page match | open | Shots: 0 | 73.9% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 101 | falcon-7bTechnology Innovation Instituteno page match | open | default | 73.8% | not dated | The Falcon Series of Open Language Models |
| 102 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 0 | 73.8% | not dated | XGen-7B Technical Report |
| 103 | LLaMA-7BMeta AIno page match | open | Shots: 0 | 73.2% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 104 | davinciOpenAI | closed | Shots: 5 | 72.2% | not dated | Stanford HELM |
| 105 | LLaMA-13BMeta AIno page match | open | Shots: 5 | 71.4% | not dated | Stanford HELM |
| 106 | open_llama_7b | not stated | Shots: 0 | 70.6% | not dated | XGen-7B Technical Report |
| 107 | bigscience/bloomEpoch: bloom, Hugging Face,BigScience | open | Shots: 5 | 70.4% | not dated | Stanford HELM |
| 108 | mpt-30bMosaicMLno page match | open | Shots: 5 | 70.4% | not dated | Stanford HELM |
| 109 | gemma-2bGoogle DeepMindno page match | open | Shots: 0 | 69.4% | not dated | Gemma: Open Models Based on Gemini Research and Technology |
| 110 | RedPajama-INCITE-7B-Base | not stated | Shots: 0 | 69.3% | not dated | XGen-7B Technical Report |
| 111 | falcon-7bTechnology Innovation Instituteno page match | open | Shots: 0 | 68.5% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 112 | Qwen/Qwen-1_8BEpoch: Qwen-1_8B | open | default | 68.0% | not dated | Qwen Technical Report |
| 113 | falcon-7bTechnology Innovation Instituteno page match | open | default | 67.5% | not dated | Qwen Technical Report |
| 114 | falcon-7bTechnology Innovation Instituteno page match | open | default | 67.5% | not dated | Llama 2: Open Foundation and Fine-Tuned Chat Models |
| 115 | Baichuan-2-13B-BaseBaichuanno page match | open | default | 67.0% | not dated | Qwen Technical Report |
| 116 | curieOpenAI | not stated | Shots: 5 | 65.6% | not dated | Stanford HELM |
| 117 | gpt-j-6bEleutherAI,LAIONno page match | open | Shots: 0 | 65.4% | not dated | XGen-7B Technical Report |
| 118 | facebook/opt-13bEpoch: opt-13b | open | Shots: 0 | 65.0% | not dated | XGen-7B Technical Report |
| 119 | EleutherAI/gpt-neox-20bEpoch: gpt-neox-20b, EleutherAI | open | Shots: 0 | 64.9% | not dated | XGen-7B Technical Report |
| 120 | internlm/internlm-7bEpoch: internlm-7b | open | default | 64.1% | not dated | Qwen Technical Report |
| 121 | Baichuan-2-7B-BaseBaichuanno page match | open | default | 63.2% | not dated | Qwen Technical Report |
| 122 | text-curie-001OpenAI | closed | Shots: 5 | 62.0% | not dated | Stanford HELM |
| 123 | gpt-neo-2.7BEleutherAIno page match | open | Shots: 0 | 61.8% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 124 | gpt2-xlOpenAIno page match | open | Shots: 0 | 61.8% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 125 | Cerebras-GPT-13BCerebras Systemsno page match | open | Shots: 0 | 61.1% | not dated | XGen-7B Technical Report |
| 126 | text-davinci-001OpenAI | closed | Shots: 0 | 60.5% | not dated | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model |
| 127 | text-davinci-001OpenAI | closed | Shots: 0 | 60.5% | not dated | LLaMA: Open and Efficient Foundation Language Models |
| 128 | text-davinci-001OpenAI | closed | default | 60.5% | not dated | The Falcon Series of Open Language Models |
| 129 | text-davinci-001OpenAI | closed | Shots: 0 | 60.5% | not dated | Training Compute-Optimal Large Language Models |
| 130 | facebook/opt-1.3bEpoch: opt-1.3b, Meta AI | open | Shots: 0 | 59.6% | not dated | Textbooks Are All You Need II: phi-1.5 technical report |
| 131 | stabilityai/stablelm-tuned-alpha-7bEpoch: stablelm-tuned-alpha-7b | open | Shots: 0 | 59.0% | not dated | XGen-7B Technical Report |
| 132 | adaOpenAI | not stated | Shots: 5 | 58.1% | not dated | Stanford HELM |
| 133 | babbageOpenAI | not stated | Shots: 5 | 57.4% | not dated | Stanford HELM |
| 134 | dolly-v2-12bDatabricksno page match | open | Shots: 0 | 56.3% | not dated | XGen-7B Technical Report |
| 135 | text-ada-001OpenAI | not stated | Shots: 5 | 46.4% | not dated | Stanford HELM |
| 136 | text-babbage-001OpenAI | closed | Shots: 5 | 45.1% | not dated | Stanford HELM |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method