Skip to content
Weights Not Walls
Menu

WeirdML v3

Reported

No description yet: we only describe an eval set when we can do it correctly.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Link
https://htihle.github.io/weirdml.html
Score shown
Epoch's column "Max performance", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
Results
27 in all: 6 for open-weight models, 21 for closed ones. 5 linked to a model page here.

Best open-weight result: deepseek-ai/DeepSeek-V4.1-Flash-1017, 19.7%, number 15 of 27 results.

WeirdML v3 results
#ModelWeightsSettingScoreDateSource
1claude-fable-5-2_xhighAnthropicclosedeffort xhigh; Agent: claude_code; Reasoning effort: xhigh65.3%not datedhtihle.github.io
2gpt-6-astra_xhighOpenAIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh54.0%not datedhtihle.github.io
3gpt-6.1-sol_xhighOpenAIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh50.6%not datedhtihle.github.io
4claude-opus-5-5_xhighAnthropicclosedeffort xhigh; Agent: claude_code; Reasoning effort: xhigh49.8%not datedhtihle.github.io
5claude-fable-5-1_xhighAnthropicclosedeffort xhigh; Agent: claude_code; Reasoning effort: xhigh43.5%not datedhtihle.github.io
6gemini-4-argon_highGoogle DeepMindclosedeffort high; Agent: gemini_cli; Reasoning effort: high41.4%not datedhtihle.github.io
7gpt-6.1-terra_xhighOpenAIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh40.4%not datedhtihle.github.io
8claude-sonnet-5-5_xhighAnthropicclosedeffort xhigh; Agent: claude_code; Reasoning effort: xhigh37.0%not datedhtihle.github.io
9claude-opus-5_xhighAnthropicclosedeffort xhigh; Agent: claude_code; Reasoning effort: xhigh32.5%not datedhtihle.github.io
10claude-fable-5_xhighAnthropicclosedeffort xhigh; Agent: claude_code; Reasoning effort: xhigh27.9%not datedhtihle.github.io
11gpt-6-sol_xhighOpenAIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh27.6%not datedhtihle.github.io
12muse-wonder-1.0_xhighMeta AIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh24.9%not datedhtihle.github.io
13claude-haiku-5-5_xhighAnthropicclosedeffort xhigh; Agent: claude_code; Reasoning effort: xhigh24.1%not datedhtihle.github.io
14gpt-5.6-sol_xhighOpenAIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh21.4%not datedhtihle.github.io
15deepseek-ai/DeepSeek-V4.1-Flash-1017Epoch: deepseek-v4.1-flash-1017_high, DeepSeekopeneffort high; Agent: codex_cli; Reasoning effort: high19.7%not datedhtihle.github.io
16moonshotai/Kimi-K3.1-ProEpoch: kimi-k3.1-pro_xhigh, Moonshotopeneffort xhigh; Agent: kimi_code; Reasoning effort: xhigh18.9%not datedhtihle.github.io
17grok-4.7_xhighxAIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh18.7%not datedhtihle.github.io
18moonshotai/Kimi-K3Epoch: kimi-k3_high, Moonshotopeneffort high; Agent: kimi_code; Reasoning effort: high15.4%not datedhtihle.github.io
19gemini-3.8-flash_highGoogle DeepMindclosedeffort high; Agent: gemini_cli; Reasoning effort: high14.6%not datedhtihle.github.io
20muse-spark-1.3_xhighMeta AIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh12.1%not datedhtihle.github.io
21gpt-6-luna_xhighOpenAIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh12.0%not datedhtihle.github.io
22mistral-large-4-0_highMistral AIno page matchopeneffort high; Agent: opencode; Reasoning effort: high11.8%not datedhtihle.github.io
23deepseek-ai/DeepSeek-V4.1-FlashEpoch: deepseek-v4.1-flash_high, DeepSeekopeneffort high; Agent: opencode; Reasoning effort: high11.6%not datedhtihle.github.io
24gpt-5.2-2025-12-11_xhighOpenAIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh9.7%not datedhtihle.github.io
25gpt-5.6-luna_xhighOpenAIclosedeffort xhigh; Agent: codex_cli; Reasoning effort: xhigh9.3%not datedhtihle.github.io
26claude-opus-4-5-20251101_highAnthropicclosedeffort high; Agent: claude_code; Reasoning effort: high7.1%not datedhtihle.github.io
27thinkingmachines/InklingEpoch: Inkling_xhigh, Thinking Machinesopeneffort xhigh; Agent: opencode; Reasoning effort: xhigh4.4%not datedhtihle.github.io

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method