Coding agent index expands to 322 tasks as two Claude models score 58

Artificial Analysis has revised its Coding Agent Index, expanding the set of tasks in which agents modify real open-source repositories from 100 to 132. The v2.1 index, announced on November 14 UTC, covers 322 tasks in all, including terminal work and questions about existing code. The evaluator announced a score of 58 for Claude Fable 5.2 and Claude Opus 5.5 in Claude Code at specified reasoning settings.

Each row describes a model running in a particular tool with a particular set of options. A development team weighing a change to one model's reasoning setting can look at the average time and API cost alongside the score. A saved score from v2.0 needs its version label, because the new table changes both part of the task set and some of the grading.

Code changes and terminal work get separate scores

FrontierCode-AA 2.0 is Artificial Analysis's implementation of a test designed by Cognition. An agent changes an open-source repository, and the result is assessed against criteria for a change a maintainer would accept. A submission that misses a mandatory requirement scores zero. If it clears those requirements, the remaining criteria determine how much credit it earns. The full set of test tasks is private.

Cognition says v2.0 of its own benchmark draws from more repositories, and individual tasks tend to require changes across more of a codebase. The grading checks whether the requested feature works and whether the patch respects the project's existing design constraints. The exercise tests whether a feature can be delivered as a change a project could merge.

Terminal-Bench 4.1 uses the same 66 tasks and environments as the previous release. Its developers changed the programs that verify the work. In one FreeCAD task, the earlier verifier could reject a component with the right dimensions when it was built using different basic shapes. The updated verifier accepts that solution. Artificial Analysis regraded saved results, retaining the earlier mark when the new verifier could not grade the output or when the difference came from variation between verifier runs.

The third test, SWE-Atlas-QnA-AA, has 124 questions about how repository code works. An answer must meet the rubric, and an attempt that changes a tracked file fails. The index averages three attempts per task, then gives each of the three tests an equal share of the final score. Two tools can therefore land near each other overall and perform quite differently on the kind of work a team does most.

Fable 5.2 scored differently across reasoning settings

Artificial Analysis reported that Claude Code with Fable 5.2 scored 58 at both the high and xhigh settings. It scored 56 at max, where more attempts ran into their time limit. Opus 5.5 at max reached 58 in Claude Code. Cognition's Devin Fusion CLI, combining Opus 5.5 with GPT-6 Luna, also appeared at 58 in the announcement.

Among the configurations scoring at least 48, the evaluator reported API costs from $1.51 to $44.22 per task. The first figure was for Codex with GPT-6.1 Sol at max, which scored 52. The second was for Claude Code with Fable 5.2 at max. Those figures are benchmark averages calculated from token usage. Consumer subscriptions for the tools have their own prices.

What a score change between versions measures

The evaluator also revised its cost calculation. It combines token counts from providers with separately measured rates at which each model can reuse cached inputs. How often an actual workload reuses parts of its prompts can change the bill. The task times shown alongside cost are averages across the tests for that configuration.

Artificial Analysis said the overall scores fell by five points on average among comparable configurations, with the FrontierCode change driving the drop. A v2.0 result and a v2.1 result reflect different coding tasks as well as different terminal grading. The leaderboard also shows the three component scores. Two configurations with a similar overall score can deliver different results on assignments such as modifying code and explaining its behavior.