Coding benchmark
NL2Repo leaderboard
Every model the catalog carries a published NL2Repo value for, ranked by that value.
CategoryCoding
MeasureRepository understanding benchmark
TasksNatural language to repository tasks
DifficultySystem-level software comprehension
Published byMiniMax M2.7: Early Echoes of Self-Evolution
A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.
NL2Repo ranking
29 models with a published NL2Repo value, ordered by that value, highest first. Models the source has not scored on this benchmark are not listed — they are unmeasured, not last.
| Rank | Model | Provider | Repository understanding benchmark |
|---|---|---|---|
| 1 |  DeepSeek V4.1 Flash | DeepSeek | 65.4 |
| 2 | DeepSeek V4 Pro 0813 | DeepSeek | 61.5 |
| 3 | OAOrnith-1.5-397B | Ornith AI | 59.5 |
| 4 |  Hy4 preview | Tencent | 58.9 |
| 5 |  GLM-5.3 | Z.AI | 58 |
| 6 | GLM-5.3-Flash | Z.AI | 56.3 |
| 7 |  Qwen3.8 Max | Alibaba | 55.9 |
| 8 | DeepSeek V4 Flash 0731 | DeepSeek | 54.2 |
| 9 | DSdots3-note Preview | Dots Studio | 49.8 |
| 10 | GLM-5.2 | Z.AI | 48.9 |
| 10 | Qwen3.8-Omni-Flash | Alibaba | 48.9 |
| 12 | DAOrnith-1.0-397B | DeepReinforce AI | 48.2 |
| 13 | Qwen3.8-Flash-Next | Alibaba | 48.1 |
| 14 | Qwen3.7 Max | Alibaba | 47.2 |
| 15 |  Seed 2.1 Pro | ByteDance | 47 |
| 16 | OAOrnith-1.5-35B-A3B | Ornith AI | 46.2 |
| 17 | Seed 2.1 Turbo | ByteDance | 43.7 |
| 18 |  Claude Opus 4.5 | Anthropic | 43.2 |
| 19 | Qwen 3.6 Max (preview) | Alibaba | 42.9 |
| 20 | GLM-5.1 | Z.AI | 42.7 |
| 21 | Qwen3.8-27B | Alibaba | 42.3 |
| 22 |  MiniMax M3 | MiniMax | 42.13 |
| 23 | Qwen3.7 Plus | Alibaba | 41.1 |
| 24 | MiniMax M2.7 | MiniMax | 39.8 |
| 25 | Qwen3.6-27B | Alibaba | 36.2 |
| 26 | DAOrnith-1.0-35B | DeepReinforce AI | 34.6 |
| 27 | OAOrnith-1.5-9B | Ornith AI | 32.4 |
| 28 | Qwen3.6-35B-A3B | Alibaba | 29.4 |
| 29 | DAOrnith-1.0-9B | DeepReinforce AI | 27.2 |
Evidence key: Observed
Rows are ordered by the value MiniMax M2.7: Early Echoes of Self-Evolution published, highest first. The source does not state whether a higher value is the better result, so this page does not either: for a benchmark that measures a rate of failure, read the table from the bottom.