Coding benchmark
DeepSWE leaderboard
Every model the catalog carries a published DeepSWE value for, ranked by that value.
CategoryCoding
MeasurePass@1 with confidence interval, cost, time, and token metadata
Tasks113 software engineering tasks across 91 repositories and 5 languages
DifficultyLong-horizon software engineering
Published byDeepSWE benchmark blog
A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.
DeepSWE ranking
28 models with a published DeepSWE value, ordered by that value, highest first. Models the source has not scored on this benchmark are not listed — they are unmeasured, not last.
| Rank | Model | Provider | Pass@1 with confidence interval, cost, time, and token metadata |
|---|---|---|---|
| 1 |  Muse Spark 1.3 | Meta | 75.4 |
| 2 |  DeepSeek V4.1 Flash | DeepSeek | 74.2 |
| 3 |  GPT-6 Astra | OpenAI | 74.1 |
| 4 | UNPareto 26.9 | Unbiased | 74.0 |
| 5 |  Gemini 3.8 Flash | 73.8 | |
| 6 |  SWE-2 | Cognition | 73.0 |
| 7 | GPT-5.6 Sol | OpenAI | 72.7 |
| 8 | GPT-5.6 Terra | OpenAI | 69.6 |
| 9 |  Claude Opus 5 | Anthropic | 68.8 |
| 10 |  Kimi K3 | Moonshot AI | 67.5 |
| 11 | Claude Fable 5.1 | Anthropic | 67.4 |
| 12 | GPT-5.6 Luna | OpenAI | 67.2 |
| 13 |  GLM-5.3 | Z.AI | 66.9 |
| 14 |  Grok 4.6 | xAI | 65.9 |
| 15 | Gemini 3.7 Flash | 65.3 | |
| 16 |  Hy4 preview | Tencent | 64.3 |
| 17 | GLM-5.3-Flash | Z.AI | 63.4 |
| 18 | DeepSeek V4 Pro 0813 | DeepSeek | 62.7 |
| 19 | Muse Spark 1.2 | Meta | 59.3 |
| 20 |  Qwen3.8-Flash-Next | Alibaba | 58.7 |
| 21 | Qwen3.8-Omni-Flash | Alibaba | 57.8 |
| 22 | Qwen3.8 Max | Alibaba | 56.6 |
| 23 | OAOrnith-1.5-397B | Ornith AI | 56.0 |
| 24 | DeepSeek V4 Flash 0731 | DeepSeek | 54.4 |
| 25 | Gemini 3.6 Flash | 49.0 | |
| 26 | Qwen3.8-27B | Alibaba | 42.2 |
| 27 |  Laguna S 2.1 | Poolside | 40.4 |
| 28 | OAOrnith-1.5-35B-A3B | Ornith AI | 22.0 |
Evidence key: Observed
Rows are ordered by the value DeepSWE benchmark blog published, highest first. The source does not state whether a higher value is the better result, so this page does not either: for a benchmark that measures a rate of failure, read the table from the bottom.