Skip to content

Research

Web and terminal access narrowed the AI score gap on Japan’s finance exams

Edited and reviewed by AI Ping Editorial Team

3-line summary

  1. Sakana AI tested 12 language models on 321 items from six Japanese finance certification exams.
  2. The gap between the top proprietary and open-weight models fell from 12.4 to 4.4 points with web and terminal access.
  3. The tax accountant exam topped out at 79.4 without tools. The authors published the evaluation code and dataset.
PracticalDevelopersOffice workers
Exam sheets, a pencil and a calculator on a wood desk
AI-generated conceptual image: Exam sheets and a calculator represent the finance certification tests behind the benchmark. · AI Ping

Japan’s Sakana AI introduced JFinBench on November 17, a test of 12 language models built from six Japanese finance certification exams. The researchers scored 321 items from the 2026 editions. The strongest proprietary model led the strongest open-weight model by 12.4 points when the systems answered unaided. That gap fell to 4.4 points when the models could use the web and a terminal.

For teams evaluating AI on Japanese tax or accounting work, the results show how performance changes when a model can look up the rules it needs. Claude Fable 5 led the unaided test. GPT-6 Astra took the top spot with tools. The benchmark graded answers against the rules applicable to the exams.

The tax exam topped out at 79.4 without tools

The six exams cover tax, accounting, financial planning, bank-counter work and bookkeeping. The researchers split 267 source questions into 321 units that could be scored separately. Of those, 193 were multiple-choice items. The others called for written answers, from a figure or term to a completed form or an essay. Some items also included images.

The overall score gives each exam equal weight. Prompts specify the exam date and the date governing which rules apply, since a change in law or accounting standards can alter the answer. The grading also differs by question type, with partial credit for individual blanks in forms and scoring criteria for essays.

At the highest reasoning setting tested without tools, Claude Fable 5 scored 95.8 overall and GPT-6 Astra scored 93.6. Kimi K3.1 Pro led the open-weight models with 83.4. Fable’s 79.4 was the best score on the tax accountant exam, where Kimi scored 48.1. The difference on that exam was larger than the gap in their overall results.

The researchers also estimated what it cost to run each model on those same questions. Without tools, Claude Fable 5 averaged $0.555 per question, GPT-6 Astra $0.248 and Kimi K3.1 Pro $0.145. Those figures apply the API list prices used in the paper to the input and output tokens consumed while answering. Grading costs are separate.

Access to reference material raised every model’s score

In the second test, models could read web pages or use a terminal for calculations. The researchers compared multiple execution setups for each model and reported its best result. GPT-6 Astra reached 97.3, Kimi K3.1 Pro reached 92.9 and Claude Fable 5 rose to 96.4. All 12 models scored higher with tools.

The authors classified failures and found that missing knowledge of Japanese rules was the most common primary cause overall. Some answers captured the purpose of a law while omitting specific requirements. In the setup shared across all models, 67 percent of attempts retrieved a web page. Japan’s government legislation site was the most frequently accessed destination.

The tool-assisted scores include the effect of both the model and the execution setup. Each result averages three runs. The test measures performance on exam tasks. Work with customer records and internal procedures would require a separate evaluation.

An open laptop beside a reference book and an exam sheet
AI-generated conceptual image: A laptop and an open reference book illustrate the test with access to outside sources. · AI Ping

Researchers checked every converted item

The team converted publicly posted exam PDFs into items that the models could answer and the benchmark could grade. A coding agent wrote the conversion code for each exam’s layout. A researcher checked all 321 items against the originals, a process the paper says took about 12 hours. The same workflow can be used when new editions of the exams appear.

A hand holding a red pencil over two sets of papers
AI-generated conceptual image: A hand comparing two sets of papers illustrates checking the converted exam questions against their originals. · AI Ping

The team published the results, data and evaluation code. The material available to download varies by exam, because the rights to redistribute questions and answer keys differ. The tax accountant questions are in the public dataset, while the authors provide a recipe for obtaining a third party’s answer material. The dataset lists the source and terms for each exam.

Sources