Why FrontierCode Reveals Top AI Models Fail at Real-World Code Mergeability

It’s Not a Model, It’s a Test

Community chatter mistakenly announced Cognition’s latest release as a new model, but FrontierCode is actually a benchmark designed to evaluate whether AI‑generated code can be merged into real projects.

13.4% – Scores Collapse When the Test Changes

The hardest “Diamond” subset contains 50 tasks. Claude Opus 4.8 achieved 13.4%, GPT‑5.5 6.3%, Gemini 3.1 Pro 4.7%, and the open‑source Kimi K2.6 only 3.8%.

These numbers contrast sharply with the ~70% pass rates the same models obtain on SWE‑bench, because the two benchmarks ask fundamentally different questions: SWE‑bench checks only whether tests pass, while FrontierCode asks whether a maintainer would actually approve the pull request.

Code that passes tests may still modify unrelated files, introduce out‑of‑style dependencies, duplicate helpers, or have weak test coverage that only hits the happy path. Such issues are penalized by FrontierCode.

The scoring rubric goes beyond correctness to include test quality, code style, scope restraint, and adherence to the codebase’s implicit conventions, simulating a senior reviewer’s line‑by‑line scrutiny.

36 Projects, 40 Hours per Question

The benchmark’s difficulty stems from its origin: 36 flagship open‑source projects (e.g., Celery, Budibase, uppy) contributed tasks, each crafted by maintainers who spent over 40 hours designing the problem and writing more than 3,000 rubric rules per task.

This is not crowdsourced labeling; it translates hidden maintainer conventions into a machine‑readable rubric, reducing the mis‑classification rate of SWE‑Bench Pro by 81% and becoming the most precise code‑quality benchmark available.

Quantifying “AI Slop” in Your Project

Many developers have seen AI‑generated code that compiles and passes CI yet feels off during maintenance: inconsistent naming, tangled abstractions, catch‑all error handling, and tests that only cover the happy path. FrontierCode quantifies this “AI slop,” showing that top models are still far from producing code that reviewers would comfortably merge.

The 13.4% figure proves the gap, and the benchmark highlights the risk of relying on overly lax self‑review standards for AI‑generated changes.

Adjusting Workflows, Not Expecting a Quick Fix

In the short term (one to two years), models are unlikely to jump to 70% mergeability. A pragmatic approach is to treat human reviewers as the gold standard.

Suggested habits include: after AI generates a PR, manually ask whether you would approve it as a teammate’s change; embed the project’s implicit conventions into a prompt context; and reject overly large diffs, asking the model to narrow changes to a single module.

These simple steps are modest but effective.

Cognition has not released the exact FrontierCode questions to avoid contaminating training data, but the benchmark is open to all model developers. Over the next year we will see a race to push the Diamond subset above 30%—the point at which AI‑assisted coding workflows might truly shift.

Finally, the article invites readers to share their own experiences of having to rewrite AI‑generated code due to style, scope, or test‑quality issues.