Gemini 4 Carbon tops MathArena at an estimated $15.70 per problem

MathArena added results for Google’s Gemini 4 Carbon and Anthropic’s Claude Fable 5.2 on Nov. 13, reshuffling its ranking of AI models for mathematical reasoning. Carbon took first place with an expected score of 85.4% at its high reasoning setting. Fable 5.2 followed at 82.8%, while the previous leader, OpenAI’s GPT-6.1 Sol, ranked third at 80.6%. MathArena ran Fable and Sol at their maximum reasoning settings. It lists Carbon’s release on Nov. 10 and Fable 5.2’s on Nov. 9.

Carbon’s expected cost for one run on one problem was $15.70, compared with $1.07 for Sol. The difference in estimated performance was 4.8 percentage points, and the estimated cost was about 14.7 times as high. For teams comparing models on research mathematics, the detailed results offer a narrower comparison too. They show what each model scored and spent on specific tests all three took.

The top line includes estimates for missing tests

MathArena draws on recent competitions, olympiads and research problems. Its protocol calls for four runs on each problem. The overall measure gives each question in the active test set equal weight and uses the mean of the completed runs where results are available. Zeroes and partial credit count toward that average.

The models have different sets of recorded results. Carbon’s page includes research problems drawn from papers and a formal proof test, while Sol has results from rounds Carbon did not take. For a question a model has yet to answer, MathArena fits a statistical estimate based on the model’s observed ability and the characteristics of the question. Its reported ranges, 85.4% plus or minus 3.0 points for Carbon and 80.6% plus or minus 2.7 points for Sol, overlap.

The cost figure also blends recorded and estimated values. MathArena weights each benchmark by its question count and excludes retired tests and Project Euler from the cost calculation. Where a cost is missing, it estimates one using the model’s costs on other tests and the relative cost of that particular benchmark. The resulting number refers to a single attempt at one problem in this test mix.

Asking for a proof of a false statement

BrokenArXiv asks models to prove plausible statements drawn from conjectures and other claims that recent papers have refuted. The solver receives the statement without being told it is false. The base grading gives zero to a model that claims to prove the false statement and three points to one that explicitly identifies it as false. Partial credit accounts for responses that change a hypothesis or acknowledge an incomplete proof.

On the same October BrokenArXiv set, Carbon scored 82.72%, Sol 80.25% and Fable 73.46%. The recorded average cost per attempt was $18.06, $1.30 and $39.78, respectively. Carbon scored 2.47 percentage points higher than Sol and cost about 13.9 times as much for this set. On the September edition, Sol scored 82.70% and Carbon 82.39%.

How the work changes the bill

MathArena’s research tests let models use their own execution tools and libraries for scientific computing. Carbon and Sol list the same base output price in their test configurations, $10 per million tokens. On the October BrokenArXiv set, Carbon averaged 563,757 output tokens and Sol averaged 89,623. How much the models generated helps account for the difference in spending on the same questions.

The gap changed on ArXivLean, a formal proof benchmark. In the September round, Carbon scored 62.26% at $45.11 per problem and Sol scored 35.85% at $1.16 per problem. The two models showed a wider gap on that task than on October’s BrokenArXiv questions.

Carbon’s public record lists BrokenArXiv results for August, September and October and ArXivLean results for September. MathArena’s detailed tables show the rounds it entered and let readers inspect the responses behind individual scores.