DeepSWE
ReportedNo description yet: we only describe an eval set when we can do it correctly.
- Kind
- Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
- Link
- https://deepswe.datacurve.ai/
- Score shown
- Epoch's column "Pass@1", as a percentage, higher is better. Error bars as the source gives them.
- Results
- 69 in all: 6 for open-weight models, 63 for closed ones. 6 linked to a model page here.
Best open-weight result: zai-org/GLM-5.3, 69.0%, number 16 of 69 results.
| # | Model | Weights | Setting | Score | Date | Source |
|---|---|---|---|---|---|---|
| 1 | gpt-6-astra_xhighOpenAI | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 74.1%71.2 to 77.0 | not dated | deepswe.datacurve.ai |
| 2 | gemini-3.8-flash_highGoogle DeepMind | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 73.8%72.4 to 75.2 | not dated | deepswe.datacurve.ai |
| 3 | claude-opus-5_maxAnthropic | closed | effort max; Harness: mini-swe-agent; Reasoning effort: max | 73.6%69.8 to 77.5 | not dated | deepswe.datacurve.ai |
| 4 | gpt-6-astra_highOpenAI | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 73.2%69.8 to 76.7 | not dated | deepswe.datacurve.ai |
| 5 | gpt-6-astra_maxOpenAI | closed | effort max; Harness: mini-swe-agent; Reasoning effort: max | 73.2%72.4 to 74.1 | not dated | deepswe.datacurve.ai |
| 6 | claude-opus-5_xhighAnthropic | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 73.2%70.1 to 76.2 | not dated | deepswe.datacurve.ai |
| 7 | claude-opus-5_highAnthropic | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 72.8%70.9 to 74.8 | not dated | deepswe.datacurve.ai |
| 8 | gpt-6-astra_mediumOpenAI | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 72.8%70.2 to 75.4 | not dated | deepswe.datacurve.ai |
| 9 | gpt-5.6-sol_maxOpenAI | closed | effort max; Harness: mini-swe-agent; Reasoning effort: max | 72.7%69.8 to 75.5 | not dated | deepswe.datacurve.ai |
| 10 | gemini-3.8-flash_mediumGoogle DeepMind | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 71.0%68.7 to 73.3 | not dated | deepswe.datacurve.ai |
| 11 | gpt-5.6-sol_xhighOpenAI | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 70.7%69.9 to 71.6 | not dated | deepswe.datacurve.ai |
| 12 | claude-fable-5_xhighAnthropic | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 69.9%66.7 to 73.2 | not dated | deepswe.datacurve.ai |
| 13 | claude-fable-5_maxAnthropic | closed | effort max; Harness: mini-swe-agent; Reasoning effort: max | 69.7%65.7 to 73.8 | not dated | deepswe.datacurve.ai |
| 14 | gpt-5.6-terra_maxOpenAI | closed | effort max; Harness: mini-swe-agent; Reasoning effort: max | 69.6%67.1 to 72.2 | not dated | deepswe.datacurve.ai |
| 15 | gpt-5.6-sol_highOpenAI | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 69.4%68.0 to 70.8 | not dated | deepswe.datacurve.ai |
| 16 | zai-org/GLM-5.3Epoch: glm-5.3_max, Z.ai (Zhipu AI) | open | effort max; Harness: mini-swe-agent; Reasoning effort: max | 69.0%65.9 to 72.0 | not dated | deepswe.datacurve.ai |
| 17 | claude-opus-5_mediumAnthropic | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 68.9%67.7 to 70.1 | not dated | deepswe.datacurve.ai |
| 18 | claude-fable-5_highAnthropic | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 68.6%67.5 to 69.7 | not dated | deepswe.datacurve.ai |
| 19 | moonshotai/Kimi-K3Epoch: kimi-k3_max, Moonshot | open | effort max; Harness: mini-swe-agent; Reasoning effort: max | 68.5%64.0 to 73.1 | not dated | deepswe.datacurve.ai |
| 20 | grok-4.6_mediumxAI | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 67.5%65.2 to 69.8 | not dated | deepswe.datacurve.ai |
| 21 | gpt-5.6-luna_maxOpenAI | closed | effort max; Harness: mini-swe-agent; Reasoning effort: max | 67.2%63.2 to 71.2 | not dated | deepswe.datacurve.ai |
| 22 | gpt-5.5_xhighOpenAI | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 67.0%60.6 to 73.5 | not dated | deepswe.datacurve.ai |
| 23 | gpt-6-astra_lowOpenAI | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 67.0%65.7 to 68.3 | not dated | deepswe.datacurve.ai |
| 24 | grok-4.6_xhighxAI | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 66.7%64.6 to 68.9 | not dated | deepswe.datacurve.ai |
| 25 | gemini-3.7-flash_mediumGoogle DeepMind | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 65.5%62.4 to 68.6 | not dated | deepswe.datacurve.ai |
| 26 | claude-fable-5_mediumAnthropic | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 65.4%60.9 to 69.8 | not dated | deepswe.datacurve.ai |
| 27 | gemini-3.7-flash_highGoogle DeepMind | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 65.3%63.5 to 67.1 | not dated | deepswe.datacurve.ai |
| 28 | grok-4.6_highxAI | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 65.2%63.7 to 66.7 | not dated | deepswe.datacurve.ai |
| 29 | gpt-5.5_highOpenAI | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 64.4%61.3 to 67.5 | not dated | deepswe.datacurve.ai |
| 30 | zai-org/GLM-5.3-FlashEpoch: glm-5.3-flash_max, Z.ai (Zhipu AI) | open | effort max; Harness: mini-swe-agent; Reasoning effort: max | 63.4%59.0 to 67.8 | not dated | deepswe.datacurve.ai |
| 31 | gpt-5.6-sol_mediumOpenAI | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 61.1%59.5 to 62.6 | not dated | deepswe.datacurve.ai |
| 32 | gpt-5.6-terra_xhighOpenAI | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 60.2%58.1 to 62.3 | not dated | deepswe.datacurve.ai |
| 33 | claude-fable-5_lowAnthropic | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 59.6%56.8 to 62.4 | not dated | deepswe.datacurve.ai |
| 34 | claude-opus-4-8_maxAnthropic | closed | effort max; Harness: mini-swe-agent; Reasoning effort: max | 59.0%57.2 to 60.7 | not dated | deepswe.datacurve.ai |
| 35 | claude-opus-5_lowAnthropic | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 58.1%55.8 to 60.5 | not dated | deepswe.datacurve.ai |
| 36 | qwen3.8-max_xhighAlibaba | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 57.5%54.8 to 60.1 | not dated | deepswe.datacurve.ai |
| 37 | gpt-5.6-luna_xhighOpenAI | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 56.9%54.7 to 59.0 | not dated | deepswe.datacurve.ai |
| 38 | muse-spark-1.2_xhighMeta AI | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 54.9%52.7 to 57.0 | not dated | deepswe.datacurve.ai |
| 39 | claude-opus-4-8_xhighAnthropic | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 54.4%50.6 to 58.1 | not dated | deepswe.datacurve.ai |
| 40 | gpt-5.5_mediumOpenAI | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 54.0%51.4 to 56.5 | not dated | deepswe.datacurve.ai |
| 41 | claude-sonnet-5_maxAnthropic | closed | effort max; Harness: mini-swe-agent; Reasoning effort: max | 53.8%49.6 to 58.1 | not dated | deepswe.datacurve.ai |
| 42 | gemini-3.7-flash_lowGoogle DeepMind | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 53.8%51.2 to 56.4 | not dated | deepswe.datacurve.ai |
| 43 | gpt-5.6-terra_highOpenAI | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 53.8%49.4 to 58.1 | not dated | deepswe.datacurve.ai |
| 44 | grok-4.5_highxAI | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 53.8%51.5 to 56.0 | not dated | deepswe.datacurve.ai |
| 45 | muse-spark-1.1Meta AI | closed | Harness: mini-swe-agent; Reasoning effort: xhigh | 53.3%50.3 to 56.4 | not dated | deepswe.datacurve.ai |
| 46 | claude-opus-4-8_highAnthropic | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 51.8%47.2 to 56.3 | not dated | deepswe.datacurve.ai |
| 47 | gpt-5.4-2026-03-05_xhighOpenAI | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 51.8%50.3 to 53.3 | not dated | deepswe.datacurve.ai |
| 48 | claude-sonnet-5_xhighAnthropic | closed | effort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh | 49.7%46.2 to 53.1 | not dated | deepswe.datacurve.ai |
| 49 | claude-opus-4-8_mediumAnthropic | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 48.7%46.4 to 50.9 | not dated | deepswe.datacurve.ai |
| 50 | claude-sonnet-5_highAnthropic | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 48.2%43.7 to 52.7 | not dated | deepswe.datacurve.ai |
| 51 | gemini-3.6-flash_highGoogle DeepMind | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 46.7%43.0 to 50.4 | not dated | deepswe.datacurve.ai |
| 52 | gpt-5.6-sol_lowOpenAI | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 45.4%43.0 to 47.7 | not dated | deepswe.datacurve.ai |
| 53 | gpt-5.6-luna_highOpenAI | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 44.2%41.3 to 47.2 | not dated | deepswe.datacurve.ai |
| 54 | zai-org/GLM-5.2Epoch: glm-5.2_max, Z.ai (Zhipu AI) | open | effort max; Harness: mini-swe-agent; Reasoning effort: max | 43.8%42.1 to 45.5 | not dated | deepswe.datacurve.ai |
| 55 | grok-4.6_lowxAI | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 41.6%39.3 to 44.0 | not dated | deepswe.datacurve.ai |
| 56 | claude-opus-4-8_lowAnthropic | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 40.8%39.3 to 42.3 | not dated | deepswe.datacurve.ai |
| 57 | claude-sonnet-5_mediumAnthropic | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 39.8%36.7 to 42.9 | not dated | deepswe.datacurve.ai |
| 58 | gemini-3.5-flash_mediumGoogle DeepMind | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 37.4%35.6 to 39.2 | not dated | deepswe.datacurve.ai |
| 59 | zai-org/GLM-5.2Epoch: glm-5.2_high, Z.ai (Zhipu AI) | open | effort high; Harness: mini-swe-agent; Reasoning effort: high | 36.3%31.5 to 41.0 | not dated | deepswe.datacurve.ai |
| 60 | gemini-3.5-flash_highGoogle DeepMind | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 36.1%32.1 to 40.0 | not dated | deepswe.datacurve.ai |
| 61 | gpt-5.6-terra_mediumOpenAI | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 35.1%31.7 to 38.5 | not dated | deepswe.datacurve.ai |
| 62 | moonshotai/Kimi-K2.7-CodeEpoch: kimi-k2.7-code, Moonshot | open | Harness: mini-swe-agent | 30.5%30.0 to 31.0 | not dated | deepswe.datacurve.ai |
| 63 | claude-sonnet-5_lowAnthropic | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 30.5%29.4 to 31.6 | not dated | deepswe.datacurve.ai |
| 64 | claude-sonnet-4-6_highAnthropic | closed | effort high; Harness: mini-swe-agent; Reasoning effort: high | 29.9%25.8 to 34.0 | not dated | deepswe.datacurve.ai |
| 65 | gpt-5.5_lowOpenAI | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 27.0%24.7 to 29.3 | not dated | deepswe.datacurve.ai |
| 66 | gpt-5.6-terra_lowOpenAI | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 24.1%23.3 to 24.8 | not dated | deepswe.datacurve.ai |
| 67 | gemini-3.1-pro-previewGoogle DeepMind | closed | Harness: mini-swe-agent; Reasoning effort: high | 11.7%10.2 to 13.2 | not dated | deepswe.datacurve.ai |
| 68 | gpt-5.6-luna_mediumOpenAI | closed | effort medium; Harness: mini-swe-agent; Reasoning effort: medium | 11.3%10.5 to 12.1 | not dated | deepswe.datacurve.ai |
| 69 | gpt-5.6-luna_lowOpenAI | closed | effort low; Harness: mini-swe-agent; Reasoning effort: low | 1.5%0.7 to 2.4 | not dated | deepswe.datacurve.ai |
Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method