Rankings
LLM coding board — models ranked for real repo work
This board ranks general-purpose LLMs for coding assistance in real repositories, not for trivia quizzes or single-function toys. Scores reflect practical use with an IDE or agent harness: multi-file edits, following project rules, and producing diffs a human can review. Vendor marketing claims are ignored.
At a glance
Scoring criteria for coding LLMs.
Leaderboard snapshot for repo coding work.
Signals that it is time to switch models.
Criteria (explicit)
Each model is scored 1–5 on:
| Criterion | What we mean |
|---|---|
| Instruction following | Honors path limits, “don’t do X,” and project skills/rules without constant reminders |
| Multi-file coherence | Changes related files consistently; avoids contradictory types/APIs across the patch |
| Diff discipline | Minimal unrelated churn; limited drive-by renames and format noise |
| Debug loop | Uses errors/test output to converge instead of thrashing random fixes |
| Repo manners | Respects existing patterns, dependencies, and architecture boundaries |
Weighting for the total (out of 10): instruction following ×2, multi-file ×2, diff discipline ×2, debug loop ×2, repo manners ×2 — each raw 1–5 is halved so the board stays on a 10-point scale (max 10).
Out of scope for this board: pure chat writing quality, image generation, latency contests without quality, and enterprise admin features.
Method notes: Rankings combine hands-on agent sessions on TypeScript/Python web repos with failure modes we see repeatedly (scope creep, fake confidence, ignored local patterns). This is an editorial scorecard, not a lab benchmark with a public leaderboard dataset.
Ranking table (2026-11-09)
| Rank | Model (typical product surface) | Score /10 | Best fit | Watch-outs |
|---|---|---|---|---|
| 1 | Claude Opus-class (IDE/agent) | 8.6 | Complex multi-file refactors with strict rules | Can over-explain; still needs path constraints |
| 2 | GPT-5-class (IDE/agent) | 8.2 | Broad stack fluency, tooling, iterative fixes | Occasional speculative files if scope is vague |
| 3 | Claude Sonnet-class | 7.8 | Fast daily PR-sized work, solid rule following | Weaker on sprawling ambiguous refactors than Opus-class |
| 4 | Gemini Pro-class (IDE/agent) | 7.2 | Large-context exploration, mixed doc+code tasks | Diff noise and boundary slips more often |
| 5 | DeepSeek / strong open-weight coding models | 6.8 | Cost-sensitive iteration, local or API bulk jobs | Needs tighter harnessing for architecture manners |
| 6 | Mid-tier “fast” assistant models | 5.5 | Small edits, boilerplate, comment/doc helpers | Unreliable ownership of multi-step repo tasks |
Treat “-class” as the current flagship tier in each family as of the update date. Exact SKU names change faster than workflows; re-check the changelog when providers rename tiers.
How to use this board
- Default daily driver: Sonnet-class or GPT-5-class for PR-sized tasks with a written constraint set.
- Hard architecture work: Opus-class when the cost of a bad boundary break is high.
- Budget lane: open-weight or mid-tier for mechanical edits after a stronger model planned the approach.
- Never: pick a model only because a social post shows a single impressive demo file.
Pair any top model with the workflow in our AI pair-programming guide: constraints first, small slices, review gates on file list and imports.
Anti-criteria (things we do not reward)
- Verbose confidence without running checks
- Rewriting working modules into new abstractions unasked
- Inventing dependencies “for cleanliness”
- Passing unit tests by weakening assertions
Retest protocol (for the next update)
When refreshing this board, keep the criteria weights fixed and swap only evidence: three PR-sized tasks (bugfix with test, feature with path limit, small refactor with “no new deps”), same repo fixtures, same review gates. Record exact model IDs, harness (IDE agent vs chat+paste), and whether project skills were enabled. Score changes should cite which criterion moved—not a vague “feels smarter.”
Changelog
- 2026-11-09 — Initial public board. Criteria locked as above. Tier labels used to survive SKU churn; next update should pin exact API/model IDs used in retests.