Coding benchmark
VulcanBench v3 leaderboard
Every model the catalog carries a published VulcanBench v3 value for, ranked by that value.
CategoryCoding
MeasurePass@1 with low, medium, and high effort
Tasks23 post-cutoff repository tasks in the v3 report
DifficultyProfessional multi-file software engineering
Published byVulcanBench
An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.
VulcanBench v3 ranking
14 models with a published VulcanBench v3 value, ordered by that value, highest first. Models the source has not scored on this benchmark are not listed — they are unmeasured, not last.
| Rank | Model | Provider | Pass@1 with low, medium, and high effort |
|---|---|---|---|
| 1 |  Grok 4.5 | xAI | 89.9 |
| 2 |  Claude Fable 5 | Anthropic | 89.5 |
| 3 |  DeepSeek V4 Flash 0731 | DeepSeek | 88.4 |
| 4 | Claude Opus 5 | Anthropic | 87.0 |
| 4 |  GPT-5.6 Sol | OpenAI | 87.0 |
| 4 | GPT-5.6 Terra | OpenAI | 87.0 |
| 4 | Grok 4.6 | xAI | 87.0 |
| 4 |  Muse Spark 1.2 | Meta | 87.0 |
| 9 | GPT-5.6 Luna | OpenAI | 85.5 |
| 10 |  Qwen3.8-27B | Alibaba | 82.6 |
| 11 | Qwen3.8 Max | Alibaba | 81.2 |
| 12 |  GLM-5.3 | Z.AI | 78.3 |
| 13 | Claude Haiku 4.5 | Anthropic | 76.2 |
| 14 |  Kimi K3 | Moonshot AI | 73.7 |
Evidence key: Observed
Rows are ordered by the value VulcanBench published, highest first. The source does not state whether a higher value is the better result, so this page does not either: for a benchmark that measures a rate of failure, read the table from the bottom.