# NL2Repo leaderboard

Coding benchmark

> Every figure below is reproduced as its upstream source published it: nothing is modelled, estimated, interpolated or converted. `Unavailable` means no source published the value — it is never a zero. Each value carries its evidence state and the date it was observed.

Page: https://modelscale.dev/leaderboards/benchmarks/coding-nl2-repo

Every model the catalog carries a published NL2Repo value for, ranked by that value.

## Benchmark record

| Field | Value |
| --- | --- |
| Category | Coding |
| Measure | Repository understanding benchmark |
| Tasks | Natural language to repository tasks |
| Difficulty | System-level software comprehension |
| Published by | [MiniMax M2.7: Early Echoes of Self-Evolution](https://www.minimax.io/news/minimax-m27-en) |

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

## NL2Repo ranking

37 models with a published NL2Repo value, ordered by that value, highest first. Models the source has not scored on this benchmark are not listed — they are unmeasured, not last.

| Rank | Model | Provider | Repository understanding benchmark | Evidence |
| --- | --- | --- | --- | --- |
| 1 | [DeepSeek V4.1 Flash 1017](https://modelscale.dev/models/deepseek-v4-1-flash) | DeepSeek | 68.2 | Observed 2026-11-21 · source benchlm:models |
| 2 | [ERNIE 5.5 Flash Preview](https://modelscale.dev/models/ernie-5-5-flash-preview) | Baidu | 64.8 | Observed 2026-11-21 · source benchlm:models |
| 3 | [DeepSeek V4 Pro](https://modelscale.dev/models/deepseek-v4-pro) | DeepSeek | 61.5 | Observed 2026-11-21 · source benchlm:models |
| 3 | [DeepSeek V4 Pro 0813](https://modelscale.dev/models/deepseek-v4-pro-max) | DeepSeek | 61.5 | Last good 2026-11-18 · source benchlm:models |
| 5 | [MiniMax M3.1 Flash](https://modelscale.dev/models/minimax-m3-1-flash) | MiniMax | 61.2 | Observed 2026-11-21 · source benchlm:models |
| 6 | [LongCat-2.5](https://modelscale.dev/models/longcat-2-5) | Meituan | 60.5 | Observed 2026-11-21 · source benchlm:models |
| 7 | [Ornith-1.5-397B](https://modelscale.dev/models/ornith-1-5-397b) | Ornith AI | 59.5 | Observed 2026-11-21 · source benchlm:models |
| 8 | [Hy4 preview](https://modelscale.dev/models/hy4-preview) | Tencent | 58.9 | Observed 2026-11-21 · source benchlm:models |
| 9 | [GLM-5.3](https://modelscale.dev/models/glm-5-3) | Z.AI | 58 | Observed 2026-11-21 · source benchlm:models |
| 10 | [GLM-5.3-Flash](https://modelscale.dev/models/glm-5-3-flash) | Z.AI | 56.3 | Observed 2026-11-21 · source benchlm:models |
| 11 | [Qwen3.8 Max](https://modelscale.dev/models/qwen3-8-max) | Alibaba | 55.9 | Observed 2026-11-21 · source benchlm:models |
| 12 | [DeepSeek V4 Flash 0731](https://modelscale.dev/models/deepseek-v4-flash-max) | DeepSeek | 54.2 | Observed 2026-11-21 · source benchlm:models |
| 13 | [dots3-note Preview](https://modelscale.dev/models/dots3-note-preview) | Dots Studio | 49.8 | Observed 2026-11-21 · source benchlm:models |
| 14 | [GLM-5.2](https://modelscale.dev/models/glm-5-2) | Z.AI | 48.9 | Observed 2026-11-21 · source benchlm:models |
| 14 | [Qwen3.8-Omni-Flash](https://modelscale.dev/models/qwen3-8-omni-flash) | Alibaba | 48.9 | Observed 2026-11-21 · source benchlm:models |
| 16 | [Ornith-1.0-397B](https://modelscale.dev/models/ornith-1-0-397b) | DeepReinforce AI | 48.2 | Observed 2026-11-21 · source benchlm:models |
| 17 | [Qwen3.8-Flash-Next](https://modelscale.dev/models/qwen3-8-flash-next) | Alibaba | 48.1 | Observed 2026-11-21 · source benchlm:models |
| 18 | [Qwen3.7 Max](https://modelscale.dev/models/qwen3-7-max) | Alibaba | 47.2 | Observed 2026-11-21 · source benchlm:models |
| 19 | [Seed 2.1 Pro](https://modelscale.dev/models/seed-2-1-pro) | ByteDance | 47 | Observed 2026-11-21 · source benchlm:models |
| 20 | [Ornith-1.5-35B-A3B](https://modelscale.dev/models/ornith-1-5-35b-a3b) | Ornith AI | 46.2 | Observed 2026-11-21 · source benchlm:models |
| 21 | [Seed 2.1 Turbo](https://modelscale.dev/models/seed-2-1-turbo) | ByteDance | 43.7 | Observed 2026-11-21 · source benchlm:models |
| 22 | [Claude Opus 4.5](https://modelscale.dev/models/claude-opus-4-5) | Anthropic | 43.2 | Observed 2026-11-21 · source benchlm:models |
| 23 | [Qwen 3.6 Max (preview)](https://modelscale.dev/models/qwen3-6-max-preview) | Alibaba | 42.9 | Observed 2026-11-21 · source benchlm:models |
| 24 | [GLM-5.1](https://modelscale.dev/models/glm-5-1) | Z.AI | 42.7 | Observed 2026-11-21 · source benchlm:models |
| 25 | [Qwen3.8-27B](https://modelscale.dev/models/qwen3-8-27b) | Alibaba | 42.3 | Observed 2026-11-21 · source benchlm:models |
| 26 | [MiniMax M3](https://modelscale.dev/models/minimax-m3) | MiniMax | 42.13 | Observed 2026-11-21 · source benchlm:models |
| 27 | [Qwen3.7 Plus](https://modelscale.dev/models/qwen3-7-plus) | Alibaba | 41.1 | Observed 2026-11-21 · source benchlm:models |
| 28 | [MiniMax M2.7](https://modelscale.dev/models/minimax-m2-7) | MiniMax | 39.8 | Observed 2026-11-21 · source benchlm:models |
| 29 | [Qwen3.6 Plus](https://modelscale.dev/models/qwen3-6-plus) | Alibaba | 37.9 | Observed 2026-11-21 · source benchlm:models |
| 30 | [Qwen3.6-27B](https://modelscale.dev/models/qwen3-6-27b) | Alibaba | 36.2 | Observed 2026-11-21 · source benchlm:models |
| 31 | [GLM-5](https://modelscale.dev/models/glm-5) | Z.AI | 35.9 | Observed 2026-11-21 · source benchlm:models |
| 32 | [Ornith-1.0-35B](https://modelscale.dev/models/ornith-1-0-35b) | DeepReinforce AI | 34.6 | Observed 2026-11-21 · source benchlm:models |
| 33 | [Ornith-1.5-9B](https://modelscale.dev/models/ornith-1-5-9b) | Ornith AI | 32.4 | Observed 2026-11-21 · source benchlm:models |
| 34 | [Qwen3.5 397B](https://modelscale.dev/models/qwen3-5-397b) | Alibaba | 32.2 | Observed 2026-11-21 · source benchlm:models |
| 35 | [Kimi K2.5](https://modelscale.dev/models/kimi-k2-5) | Moonshot AI | 32 | Observed 2026-11-21 · source benchlm:models |
| 36 | [Qwen3.6-35B-A3B](https://modelscale.dev/models/qwen3-6-35b-a3b) | Alibaba | 29.4 | Observed 2026-11-21 · source benchlm:models |
| 37 | [Ornith-1.0-9B](https://modelscale.dev/models/ornith-1-0-9b) | DeepReinforce AI | 27.2 | Observed 2026-11-21 · source benchlm:models |

Rows are ordered by the value the source published, highest first. The source does not state whether a higher value is the better result, so this document does not either: for a benchmark that measures a rate of failure, read the table from the bottom.
