Mo finds more bugs in Momentic’s QA benchmark, at a higher cost

Momentic published QA Bench on November 2, comparing its Mo testing agent with Codex, Claude Code and Devin on running web applications. Under the company’s 60-minute condition, Mo matched an average of 269.3 reference bugs, against 86.3 for Codex using GPT-6.1 Sol with Playwright MCP—about 3.1 times as many.

That extra coverage came at a higher cost. Mo cost $2.67 per matched bug, compared with $0.80 for that Codex configuration. Precision—the share of reported findings that matched real reference bugs—was 84.2% and 86.0%, respectively. Mo’s recall, the share of the reference set it found, was 19.3%. The precision figure therefore cannot be read as finding 84% of all known bugs.

The benchmark covers 20 product areas across Gitea, Metabase, Twenty and Saleor: code hosting, analytics, customer management and e-commerce administration. Each run starts in a fresh environment with seeded data and an area-specific brief. Agents explore the interface and report differences between expected and observed behavior.

The 60-minute allowance applies to each product area, rather than the entire suite. Most configurations ran each area three times; Devin ran it once. Published match counts add the per-area means, which explains their fractional values. Environment preparation and grading are outside the exploration time budget.

The dataset contains 1,396 human-reviewed bugs and 501 rejected reports. Its defects occur in the pinned application versions; none were planted for the benchmark. Candidate labels originated in exploratory runs by Mo, Codex and Claude Code, however, so the reference set can share their blind spots. Recall measures coverage of reviewed labels, not every defect an application contains.

More findings, with evidence to review

A GPT-6.1 Sol judge maps submitted findings to the reference lists. Previously unclassified reports enter a human review queue, where they become bugs or rejected claims. Repeating one bug produces one match and a separate duplicate entry. Momentic designed, ran and graded the evaluation while also developing Mo.

The published runs include exceptions to the intended access restrictions. Although agents were supposed to inspect the application without reading its source or answer lists, Codex web search remained enabled. Some browser-tool configurations also retained shell access to the external web. The repository documents an optional restriction on answer searching and labels runs using it separately from the existing results.

In the product itself, Momentic says suspected functional defects must be reproduced by a separate agent from a clean browser or device session before becoming confirmed bugs. Unreproduced suspicions are excluded from confirmed findings; static issues such as typos use screenshots. Parallel agents receive separate browsers or devices, but they may still share accounts and backend data.

Session reports retain expected and actual behavior, reproduction steps and evidence. They also distinguish completed checks from blocked work caused by access or missing data. Exported evidence differentiates unavailable logs from logs that were read but contained no matching entries. An export cannot restore evidence that was never captured. Exporting an unfinished session returns what has been saved so far, with a warning that the report may be partial.

Mo requires a running target, a brief and any necessary test account. Web sessions use hosted browsers, iOS uses remote simulators and Android uses remote emulators; mobile sessions require workspace access. Each session covers one target, with separate sessions needed to cover the web, iOS and Android versions of an application. Operating-system dialogs or flows that open another desktop application can block browser-based verification, leaving a coverage gap in the report.