WeirdML v3
ReportedNo description yet: we only describe an eval set when we can do it correctly.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Link
- https://htihle.github.io/weirdml.html
- Score shown
- Epoch's column "Max performance", as a percentage, higher is better; random guessing scores about 0.0%. The source gives no error bars.
- Results
- 27 in all: 6 for open-weight models, 21 for closed ones. 5 linked to a model page here.
Best open-weight result: deepseek-ai/DeepSeek-V4.1-Flash-1017, 19.7%, number 15 of 27 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | claude-fable-5-2_xhighAnthropic | closed | effort xhigh; Agent: claude_code; Reasoning effort: xhigh | 65.3% | not dated | htihle.github.io |
| 2 | gpt-6-astra_xhighOpenAI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 54.0% | not dated | htihle.github.io |
| 3 | gpt-6.1-sol_xhighOpenAI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 50.6% | not dated | htihle.github.io |
| 4 | claude-opus-5-5_xhighAnthropic | closed | effort xhigh; Agent: claude_code; Reasoning effort: xhigh | 49.8% | not dated | htihle.github.io |
| 5 | claude-fable-5-1_xhighAnthropic | closed | effort xhigh; Agent: claude_code; Reasoning effort: xhigh | 43.5% | not dated | htihle.github.io |
| 6 | gemini-4-argon_highGoogle DeepMind | closed | effort high; Agent: gemini_cli; Reasoning effort: high | 41.4% | not dated | htihle.github.io |
| 7 | gpt-6.1-terra_xhighOpenAI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 40.4% | not dated | htihle.github.io |
| 8 | claude-sonnet-5-5_xhighAnthropic | closed | effort xhigh; Agent: claude_code; Reasoning effort: xhigh | 37.0% | not dated | htihle.github.io |
| 9 | claude-opus-5_xhighAnthropic | closed | effort xhigh; Agent: claude_code; Reasoning effort: xhigh | 32.5% | not dated | htihle.github.io |
| 10 | claude-fable-5_xhighAnthropic | closed | effort xhigh; Agent: claude_code; Reasoning effort: xhigh | 27.9% | not dated | htihle.github.io |
| 11 | gpt-6-sol_xhighOpenAI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 27.6% | not dated | htihle.github.io |
| 12 | muse-wonder-1.0_xhighMeta AI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 24.9% | not dated | htihle.github.io |
| 13 | claude-haiku-5-5_xhighAnthropic | closed | effort xhigh; Agent: claude_code; Reasoning effort: xhigh | 24.1% | not dated | htihle.github.io |
| 14 | gpt-5.6-sol_xhighOpenAI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 21.4% | not dated | htihle.github.io |
| 15 | deepseek-ai/DeepSeek-V4.1-Flash-1017Epoch: deepseek-v4.1-flash-1017_high, DeepSeek | open | effort high; Agent: codex_cli; Reasoning effort: high | 19.7% | not dated | htihle.github.io |
| 16 | moonshotai/Kimi-K3.1-ProEpoch: kimi-k3.1-pro_xhigh, Moonshot | open | effort xhigh; Agent: kimi_code; Reasoning effort: xhigh | 18.9% | not dated | htihle.github.io |
| 17 | grok-4.7_xhighxAI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 18.7% | not dated | htihle.github.io |
| 18 | moonshotai/Kimi-K3Epoch: kimi-k3_high, Moonshot | open | effort high; Agent: kimi_code; Reasoning effort: high | 15.4% | not dated | htihle.github.io |
| 19 | gemini-3.8-flash_highGoogle DeepMind | closed | effort high; Agent: gemini_cli; Reasoning effort: high | 14.6% | not dated | htihle.github.io |
| 20 | muse-spark-1.3_xhighMeta AI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 12.1% | not dated | htihle.github.io |
| 21 | gpt-6-luna_xhighOpenAI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 12.0% | not dated | htihle.github.io |
| 22 | mistral-large-4-0_highMistral AIno page match | open | effort high; Agent: opencode; Reasoning effort: high | 11.8% | not dated | htihle.github.io |
| 23 | deepseek-ai/DeepSeek-V4.1-FlashEpoch: deepseek-v4.1-flash_high, DeepSeek | open | effort high; Agent: opencode; Reasoning effort: high | 11.6% | not dated | htihle.github.io |
| 24 | gpt-5.2-2025-12-11_xhighOpenAI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 9.7% | not dated | htihle.github.io |
| 25 | gpt-5.6-luna_xhighOpenAI | closed | effort xhigh; Agent: codex_cli; Reasoning effort: xhigh | 9.3% | not dated | htihle.github.io |
| 26 | claude-opus-4-5-20251101_highAnthropic | closed | effort high; Agent: claude_code; Reasoning effort: high | 7.1% | not dated | htihle.github.io |
| 27 | thinkingmachines/InklingEpoch: Inkling_xhigh, Thinking Machines | open | effort xhigh; Agent: opencode; Reasoning effort: xhigh | 4.4% | not dated | htihle.github.io |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method