Skip to content
Weights Not Walls
Menu

DeepSWE

Reported

No description yet: we only describe an eval set when we can do it correctly.

Kind
Published by a third party's leaderboard, collected by Epoch AI. The leaderboard is named and linked.
Link
https://deepswe.datacurve.ai/
Score shown
Epoch's column "Pass@1", as a percentage, higher is better. Error bars as the source gives them.
Results
69 in all: 6 for open-weight models, 63 for closed ones. 6 linked to a model page here.

Best open-weight result: zai-org/GLM-5.3, 69.0%, number 16 of 69 results.

DeepSWE results
#ModelWeightsSettingScoreDateSource
1gpt-6-astra_xhighOpenAIclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh74.1%71.2 to 77.0not dateddeepswe.datacurve.ai
2gemini-3.8-flash_highGoogle DeepMindclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high73.8%72.4 to 75.2not dateddeepswe.datacurve.ai
3claude-opus-5_maxAnthropicclosedeffort max; Harness: mini-swe-agent; Reasoning effort: max73.6%69.8 to 77.5not dateddeepswe.datacurve.ai
4gpt-6-astra_highOpenAIclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high73.2%69.8 to 76.7not dateddeepswe.datacurve.ai
5gpt-6-astra_maxOpenAIclosedeffort max; Harness: mini-swe-agent; Reasoning effort: max73.2%72.4 to 74.1not dateddeepswe.datacurve.ai
6claude-opus-5_xhighAnthropicclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh73.2%70.1 to 76.2not dateddeepswe.datacurve.ai
7claude-opus-5_highAnthropicclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high72.8%70.9 to 74.8not dateddeepswe.datacurve.ai
8gpt-6-astra_mediumOpenAIclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium72.8%70.2 to 75.4not dateddeepswe.datacurve.ai
9gpt-5.6-sol_maxOpenAIclosedeffort max; Harness: mini-swe-agent; Reasoning effort: max72.7%69.8 to 75.5not dateddeepswe.datacurve.ai
10gemini-3.8-flash_mediumGoogle DeepMindclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium71.0%68.7 to 73.3not dateddeepswe.datacurve.ai
11gpt-5.6-sol_xhighOpenAIclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh70.7%69.9 to 71.6not dateddeepswe.datacurve.ai
12claude-fable-5_xhighAnthropicclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh69.9%66.7 to 73.2not dateddeepswe.datacurve.ai
13claude-fable-5_maxAnthropicclosedeffort max; Harness: mini-swe-agent; Reasoning effort: max69.7%65.7 to 73.8not dateddeepswe.datacurve.ai
14gpt-5.6-terra_maxOpenAIclosedeffort max; Harness: mini-swe-agent; Reasoning effort: max69.6%67.1 to 72.2not dateddeepswe.datacurve.ai
15gpt-5.6-sol_highOpenAIclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high69.4%68.0 to 70.8not dateddeepswe.datacurve.ai
16zai-org/GLM-5.3Epoch: glm-5.3_max, Z.ai (Zhipu AI)openeffort max; Harness: mini-swe-agent; Reasoning effort: max69.0%65.9 to 72.0not dateddeepswe.datacurve.ai
17claude-opus-5_mediumAnthropicclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium68.9%67.7 to 70.1not dateddeepswe.datacurve.ai
18claude-fable-5_highAnthropicclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high68.6%67.5 to 69.7not dateddeepswe.datacurve.ai
19moonshotai/Kimi-K3Epoch: kimi-k3_max, Moonshotopeneffort max; Harness: mini-swe-agent; Reasoning effort: max68.5%64.0 to 73.1not dateddeepswe.datacurve.ai
20grok-4.6_mediumxAIclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium67.5%65.2 to 69.8not dateddeepswe.datacurve.ai
21gpt-5.6-luna_maxOpenAIclosedeffort max; Harness: mini-swe-agent; Reasoning effort: max67.2%63.2 to 71.2not dateddeepswe.datacurve.ai
22gpt-5.5_xhighOpenAIclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh67.0%60.6 to 73.5not dateddeepswe.datacurve.ai
23gpt-6-astra_lowOpenAIclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low67.0%65.7 to 68.3not dateddeepswe.datacurve.ai
24grok-4.6_xhighxAIclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh66.7%64.6 to 68.9not dateddeepswe.datacurve.ai
25gemini-3.7-flash_mediumGoogle DeepMindclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium65.5%62.4 to 68.6not dateddeepswe.datacurve.ai
26claude-fable-5_mediumAnthropicclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium65.4%60.9 to 69.8not dateddeepswe.datacurve.ai
27gemini-3.7-flash_highGoogle DeepMindclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high65.3%63.5 to 67.1not dateddeepswe.datacurve.ai
28grok-4.6_highxAIclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high65.2%63.7 to 66.7not dateddeepswe.datacurve.ai
29gpt-5.5_highOpenAIclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high64.4%61.3 to 67.5not dateddeepswe.datacurve.ai
30zai-org/GLM-5.3-FlashEpoch: glm-5.3-flash_max, Z.ai (Zhipu AI)openeffort max; Harness: mini-swe-agent; Reasoning effort: max63.4%59.0 to 67.8not dateddeepswe.datacurve.ai
31gpt-5.6-sol_mediumOpenAIclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium61.1%59.5 to 62.6not dateddeepswe.datacurve.ai
32gpt-5.6-terra_xhighOpenAIclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh60.2%58.1 to 62.3not dateddeepswe.datacurve.ai
33claude-fable-5_lowAnthropicclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low59.6%56.8 to 62.4not dateddeepswe.datacurve.ai
34claude-opus-4-8_maxAnthropicclosedeffort max; Harness: mini-swe-agent; Reasoning effort: max59.0%57.2 to 60.7not dateddeepswe.datacurve.ai
35claude-opus-5_lowAnthropicclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low58.1%55.8 to 60.5not dateddeepswe.datacurve.ai
36qwen3.8-max_xhighAlibabaclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh57.5%54.8 to 60.1not dateddeepswe.datacurve.ai
37gpt-5.6-luna_xhighOpenAIclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh56.9%54.7 to 59.0not dateddeepswe.datacurve.ai
38muse-spark-1.2_xhighMeta AIclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh54.9%52.7 to 57.0not dateddeepswe.datacurve.ai
39claude-opus-4-8_xhighAnthropicclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh54.4%50.6 to 58.1not dateddeepswe.datacurve.ai
40gpt-5.5_mediumOpenAIclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium54.0%51.4 to 56.5not dateddeepswe.datacurve.ai
41claude-sonnet-5_maxAnthropicclosedeffort max; Harness: mini-swe-agent; Reasoning effort: max53.8%49.6 to 58.1not dateddeepswe.datacurve.ai
42gemini-3.7-flash_lowGoogle DeepMindclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low53.8%51.2 to 56.4not dateddeepswe.datacurve.ai
43gpt-5.6-terra_highOpenAIclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high53.8%49.4 to 58.1not dateddeepswe.datacurve.ai
44grok-4.5_highxAIclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high53.8%51.5 to 56.0not dateddeepswe.datacurve.ai
45muse-spark-1.1Meta AIclosedHarness: mini-swe-agent; Reasoning effort: xhigh53.3%50.3 to 56.4not dateddeepswe.datacurve.ai
46claude-opus-4-8_highAnthropicclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high51.8%47.2 to 56.3not dateddeepswe.datacurve.ai
47gpt-5.4-2026-03-05_xhighOpenAIclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh51.8%50.3 to 53.3not dateddeepswe.datacurve.ai
48claude-sonnet-5_xhighAnthropicclosedeffort xhigh; Harness: mini-swe-agent; Reasoning effort: xhigh49.7%46.2 to 53.1not dateddeepswe.datacurve.ai
49claude-opus-4-8_mediumAnthropicclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium48.7%46.4 to 50.9not dateddeepswe.datacurve.ai
50claude-sonnet-5_highAnthropicclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high48.2%43.7 to 52.7not dateddeepswe.datacurve.ai
51gemini-3.6-flash_highGoogle DeepMindclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high46.7%43.0 to 50.4not dateddeepswe.datacurve.ai
52gpt-5.6-sol_lowOpenAIclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low45.4%43.0 to 47.7not dateddeepswe.datacurve.ai
53gpt-5.6-luna_highOpenAIclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high44.2%41.3 to 47.2not dateddeepswe.datacurve.ai
54zai-org/GLM-5.2Epoch: glm-5.2_max, Z.ai (Zhipu AI)openeffort max; Harness: mini-swe-agent; Reasoning effort: max43.8%42.1 to 45.5not dateddeepswe.datacurve.ai
55grok-4.6_lowxAIclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low41.6%39.3 to 44.0not dateddeepswe.datacurve.ai
56claude-opus-4-8_lowAnthropicclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low40.8%39.3 to 42.3not dateddeepswe.datacurve.ai
57claude-sonnet-5_mediumAnthropicclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium39.8%36.7 to 42.9not dateddeepswe.datacurve.ai
58gemini-3.5-flash_mediumGoogle DeepMindclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium37.4%35.6 to 39.2not dateddeepswe.datacurve.ai
59zai-org/GLM-5.2Epoch: glm-5.2_high, Z.ai (Zhipu AI)openeffort high; Harness: mini-swe-agent; Reasoning effort: high36.3%31.5 to 41.0not dateddeepswe.datacurve.ai
60gemini-3.5-flash_highGoogle DeepMindclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high36.1%32.1 to 40.0not dateddeepswe.datacurve.ai
61gpt-5.6-terra_mediumOpenAIclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium35.1%31.7 to 38.5not dateddeepswe.datacurve.ai
62moonshotai/Kimi-K2.7-CodeEpoch: kimi-k2.7-code, MoonshotopenHarness: mini-swe-agent30.5%30.0 to 31.0not dateddeepswe.datacurve.ai
63claude-sonnet-5_lowAnthropicclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low30.5%29.4 to 31.6not dateddeepswe.datacurve.ai
64claude-sonnet-4-6_highAnthropicclosedeffort high; Harness: mini-swe-agent; Reasoning effort: high29.9%25.8 to 34.0not dateddeepswe.datacurve.ai
65gpt-5.5_lowOpenAIclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low27.0%24.7 to 29.3not dateddeepswe.datacurve.ai
66gpt-5.6-terra_lowOpenAIclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low24.1%23.3 to 24.8not dateddeepswe.datacurve.ai
67gemini-3.1-pro-previewGoogle DeepMindclosedHarness: mini-swe-agent; Reasoning effort: high11.7%10.2 to 13.2not dateddeepswe.datacurve.ai
68gpt-5.6-luna_mediumOpenAIclosedeffort medium; Harness: mini-swe-agent; Reasoning effort: medium11.3%10.5 to 12.1not dateddeepswe.datacurve.ai
69gpt-5.6-luna_lowOpenAIclosedeffort low; Harness: mini-swe-agent; Reasoning effort: low1.5%0.7 to 2.4not dateddeepswe.datacurve.ai

Open-weight rows are shaded. Dates are when the result was run or evaluated, where the data says. Data from Epoch AI (CC-BY 4.0), fetched 19 Nov 2026: Epoch AI, ‘Capabilities & benchmarking’. Published online at epoch.ai. Retrieved from ‘https://epoch.ai/benchmarks’ [online resource]. Epoch AI. Method