Deepnote’s Slack test finds AI agents stay quiet only 63% of the time they should

Deepnote released its Agent Initiative Benchmark on Nov. 13 to measure when an AI agent speaks up in workplace conversations. Twelve models each faced the same 50 situations at a simulated company. Across the published results, the models sent a message in 434 of 480 runs (90.4%) where help was expected. They stayed quiet in 75 of 120 runs (62.5%) where silence was appropriate.

Those figures answer separate questions for teams testing an agent in Slack. Can it spot a problem that employees have missed, and can it leave a settled conversation alone? The benchmark records whether a message went out and grades its usefulness separately. Deepnote also released the simulated company’s dataset and the code to run a set of examples, giving other teams a way to test those decisions.

Two teams working from different dates

In one illustration from Deepnote, sales expects a Friday launch while engineering has moved it back a week. An agent that sees only one team’s conversation may miss the discrepancy. Checking both schedules could give it enough evidence to alert the people making commitments to customers. The test starts from that sort of workplace context, with an agent expected to decide for itself whether anyone needs a message.

The fictional software company, PageMantle, has six years of linked records across 11 simulated services. A discussion in Slack can connect to an issue in Linear, notes in Notion or customer and billing records in HubSpot and Stripe. Most starting conversations come from one month of the company’s Slack history. The test attaches case-specific records to a shared baseline, so each model sees the same evidence for a given situation.

Sending a message and getting it right

Deepnote gave each model the same instructions, search tools and high reasoning setting. During a run, the model could look up records and post in a channel, reply to a thread, send a direct message or stay quiet. The model had a budget of 180 seconds of response time. Tool execution ran outside that budget.

Forty of the 50 situations called for help, and ten called for silence. The response count shows whether the model sent something, including messages that may have missed the point. For the separate ten-point score, two evaluator models saw the run log, company data and a reference solution with the tested model’s identity hidden. They considered whether it should have intervened, how well its message fit the evidence, how it read and where it was sent.

Where timing and cost enter the results

Gemini 4 Argon stayed silent in all ten cases that called for silence. It also sent a message in just 23 of the 40 situations where one was expected. It reached the time limit in 27 of its 50 runs. On the measure of whether a message went out, a run that ends in a timeout without posting looks the same as a decision to hold back.

GPT-6 Astra averaged 8.97 out of 10 and GPT-6.1 Sol averaged 8.65. Deepnote reported overlapping uncertainty intervals when it resampled the test cases, leaving the difference unresolved on this set. The study recorded about 54 seconds of mean model response time for each. Model charges averaged $0.693 per run for Astra and $0.112 for Sol. Those amounts cover model usage recorded during the study.

Data and examples for running the test

Deepnote’s repository includes eight runnable examples, one for each task type. The tools read a local copy of the dataset, so teams can run the examples without connecting accounts on any of the 11 services. The English-language PageMantle dataset is available on Hugging Face under CC BY 4.0. The test code is on GitHub under Apache 2.0.

Each run starts at a point chosen by the researchers, and the simulated colleagues send no replies. That design captures the first intervention in a conversation. It leaves follow-up behavior for other tests, such as whether an agent would correct its own alert after a launch date changes again or how many interruptions a team would receive over a month.