Method
How we label scores, which models we list, how we work out whether a model runs on your machine, and what we do not do yet.
Three kinds of score
- Claimed
- Reported by the model's maker, in the model card or paper.
- Reported
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Re-run by Epoch AI
- Run independently by Epoch AI, with public logs. Our own re-runs are not built yet.
Every score on the site carries one of these badges, the link to its source and, where the data gives them, the date it was run and an error bar. Scores of different kinds are never averaged, merged or ranked together. A model's own claimed scores come from the machine-readable part of its model card (model-index); many cards give scores only in their text, which we do not read.
In phase 1 the independent scores are Epoch AI's. Files in Epoch's data without _external in the name are Epoch's own runs, with logs: we label them "Re-run by Epoch AI". Files ending _external are results Epoch collected from other leaderboards, each row with its source: we label them "Reported". We have not run any evaluation ourselves yet, and the site says so wherever it matters.
Error bars: a standard error is shown as plus or minus, in percentage points; an interval as its low and high ends. Where an error figure in the data does not fit the score's scale (for example a standard error of 4.9 beside a score of 0.475) we leave the error bar out and say so, rather than guess its unit. A result's date is when it was run or evaluated; Epoch's model release dates are not used as result dates.
Several rows for one model on one set are different settings, usually the reasoning effort Epoch's name for the run ends in (_high, _max). Where a page needs one number per model (Will it run, the best open model on a set), it takes the best setting and says how many there were.
Matching scores to models
Epoch names models its own way (qwen3.8-27b_xhigh). A score attached to the wrong model is the worst mistake this site could make, so a name is linked to a Hub repo only when all of these hold: after removing provider prefixes and effort suffixes, a repo of exactly that name exists in an organisation on our makers list; Epoch's organisation for the model is that maker; Epoch's release date is within 120 days of the repo's creation; and no instruct or chat sibling could equally be meant. Anything less certain is left unlinked. The matches are in a file a person can check and correct, and every score on a model page shows Epoch's name beside it.
Closed models stay in the tables as reference rows, so you can see how an open model compares, but they never get a model page. Rows from Artificial Analysis and LMArena are in Epoch's data and are not copied here: we link to Artificial Analysis and the Arena leaderboard instead.
What we index
Models on the Hugging Face Hub tagged text-generation or image-text-to-text. Most current open language models accept images and carry the second tag, so leaving it out would leave out most new releases. The snapshot holds the top 2,000 by downloads and the top 500 by trending score for each tag, the 500 newest repos from each maker on our list, and every model an API provider serves on Hugging Face Inference Providers.
Quantised conversions (GGUF, MLX, AWQ, FP8 and similar repos) are not listed as models: they appear on their base model's page under Run it yourself. Fine-tunes are listed and say what they are a fine-tune of. Gated repos are kept and marked. Any other Hub model opens at the same address (/models/organisation/name), fetched live and marked unreviewed.
Snapshot of 19 Nov 2026: 4,715 models, 390 reviewed.
Reviewed and unreviewed
A model is reviewed when all of these hold:
- it is the maker's own repo, from an organisation on our list of original makers (kept by hand, from the organisations behind the most downloaded, most liked and trending base models);
- it is a base model or an official instruct or chat model, not a conversion, adapter or merge;
- its licence is one we have read and classed;
- the Hub gives a parameter count we can trust;
- it is not excluded by hand (each exclusion has a written reason);
- and it is in use: at least 100,000 downloads or 500 likes, or created in the last 21 days with at least 20 likes, or served by an API provider.
Everything else is unreviewed: shown, searchable and clearly marked, with only what the Hub data supports, and each unreviewed page says which rules it did not meet. The reviewed list is checked against this rule; a person still has to pass over it before launch.
Will it run: memory and verdicts
memory needed = weights + context cache + 0.5 GB
- Weights are the real file size where we read the repo on the Hub (every part of a split file added up), labelled "file size". Without a file, they are estimated as parameters times bits per weight divided by 8, labelled "estimated". The original weights row uses the size from the repo's safetensors header.
- Context cache is the key and value cache at 8k tokens unless you choose another length: 2 x key-value heads x head size x 2 bytes, per token, per layer, worked out from the model's config. Sliding-window layers are capped at their window; linear-attention and Mamba layers are not counted; latent attention counts the cached latent. Where the config is unclear every layer is counted, which overestimates. Without a config we use a rough allowance by model size and say so. The cache is counted at 16-bit; runtimes that quantise the cache need less.
- 0.5 GB for the runtime's buffers. A vision model's projector file, needed only for image input, is not counted and is listed beside the files.
Usable memory:
- graphics card: its memory less 0.5 GB for the driver and display;
- Apple silicon Mac: 70% of unified memory, since macOS keeps the rest for itself by default;
- CPU only: 80% of RAM. It works, slowly.
Verdicts, on purpose cautious:
- fits: needed is at most 90% of usable;
- tight: at most all of usable;
- needs offload to RAM: graphics cards only, when it fits in the card plus 80% of system RAM; much slower;
- does not fit: otherwise.
In the catalogue, on the home page and on Will it run, a model is judged by the largest format that fits, and "fits my machine" counts only formats of about 4 bits per weight or more (Q4_K_M, IQ4_XS, MXFP4 and larger), so a model does not count as fitting because a 2-bit version squeezes in. The format is always shown. MLX files run only on Apple silicon. A verdict says nothing about speed.
GB here means 230 bytes, as operating systems and graphics card tools report memory. The Hub shows file sizes in 109 bytes, so our sizes look about 7% smaller than the Hub's for the same file.
Bits per weight used for estimates
The median measured over every GGUF file of that level in our snapshot, where there are at least three; otherwise the format's block size.
| Level | Bits per weight | Basis |
|---|---|---|
| Q2_K | 3.12 | measured median, 611 files |
| Q3_K_M | 3.92 | measured median, 591 files |
| IQ4_XS | 4.34 | measured median, 463 files |
| Q4_K_M | 4.88 | measured median, 784 files, used for estimates |
| Q5_K_M | 5.71 | measured median, 633 files, used for estimates |
| Q6_K | 6.57 | measured median, 692 files, used for estimates |
| Q8_0 | 8.50 | measured median, 717 files, used for estimates |
| BF16 | 16.00 | measured median, 273 files, used for estimates |
Providers
Prices, context length, precision, speed, first-token latency, uptime and tool support come from the Hugging Face router's model list, as it reports them, with the time we fetched it. Prices are US dollars per million tokens. Where a provider serves a quantised copy of a model, the page says which repo it serves. We do not test providers' quality yet; precision is the best sign of it for now.
Known limits
- The data is a snapshot, refreshed by hand for now. A release from today may be missing until the next refresh; models outside the snapshot still open, unreviewed.
- Dates are when the repo was created on the Hub, which can be before the model was made public.
- Memory needs are worked out, not measured on real machines. Runtimes differ; long contexts, batch sizes and other programs on the machine change the picture.
- Independent scores cover the models Epoch AI has run or collected, mostly large and recent ones. Small models often have none.
- Model cards often give scores only in their text; those are not shown as claimed scores.
- Licence classes are our reading, in plain words, not legal advice. Read the licence.
- We have run no evaluations of our own, and no per-provider quality tests, yet.
Sources and credits
- Model metadata, file sizes and claimed scores: the Hugging Face Hub, fetched 19 Nov 2026.
- Providers: Hugging Face Inference Providers, fetched 19 Nov 2026.
- Independent scores: Epoch AI, used under CC-BY 4.0, fetched 19 Nov 2026. Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource].