Measure an AI pilot and decide when to stop

1. 1

   A fast draft may still take a long time to finish

   Suppose a report used to take 20 minutes and an AI draft appears in two. If checking takes eight more minutes and repair takes six, completion takes 16—not two. These are invented numbers. Compare the point at which the same quality standard is met.

   A baseline is the starting method used for comparison. Match task type, difficulty, scope and completion criteria before comparing times. Distinguish active human work from elapsed waiting time instead of adding incompatible measurements.

   Design work includes revisions, sales work includes proposal checks, research includes source verification and development includes testing and repair. Also account for work shifted to someone else; a fast draft can create a slow review queue.

   Here AI is an analytical assistant for organising time records and spotting overclaims, not a device that measures work automatically. Without observed start and finish records, an attractive table remains an estimate. The fictional pack lets you learn the arithmetic and reporting boundaries without using a service.

   Completion must mean the same thing in both methods. Comparing a checked baseline report with an unread new draft changes the task. Add input/draft, review and repair, then keep quality status separate instead of combining it with speed in one score.

1. 2

   Check units and totals before interpreting the result

   The table contains five invented document tasks. Values are minutes of active human work. Input/draft includes preparation, requesting and handling the draft. Waiting time and tool charges are excluded; state that scope before interpreting the numbers. The downloadable CSV matches the table.

   

   Baseline time totals 100 minutes. New-process row totals are 16, 15, 30, 12 and 20: 93 minutes overall. The difference is seven minutes, or 7% of this fictional baseline. Task C takes five minutes longer, which an aggregate alone conceals.

   Do not convert this into a claim that employee productivity increased by 7%. Five invented records omit broader costs, quality differences and learning effects. A time record also does not prove that the output met the rubric; check quality separately.

   Supply column definitions with the table. A’s two-minute input/draft value is not completion time: eight review and six repair minutes also count. The unit is active human minutes. Multiple people working simultaneously could have a different elapsed duration, but this pack contains no concurrency records.

   Recalculating in a sheet or calculator checks the assistant’s analysis. Establish 100 and 93 before requesting a narrative. Ask for row equations and the denominator rather than prescribing “the answer must be 7%.” A mismatch is a reason to locate an omitted column or row, not force agreement.

   Fictional practice records · active human minutes

   Task	Baseline	Input/draft	Review	Repair
   A	20	2	8	6
   B	18	3	8	4
   C	25	4	12	14
   D	17	2	7	3
   E	20	3	9	8
   Total	100	14	44	35

   Download fictional time records (CSV)

2. 3

   Do not let speed hide a consequential error

   A faster proposal is unusable if it promises a nonexistent capability. Put rubric results beside time records. A failed task that ends quickly is not an efficiently completed task. Keep failures and repair costs visible.

   Define a stop condition before seeing results. For this exercise, an unsupported customer promise stops external use until the review process is fixed. This is an instructional condition, not a universal numerical standard; real criteria depend on consequences and responsibilities.

   Investigate why C needed extensive repair: conflicting evidence, excessive scope or a missing check. Distinguish an observed failure from a hypothesis about its cause. Change one relevant condition and test new cases instead of assuming a different product will solve it.

   Analyse only fictional rows A–E and the column definitions. All values are active human minutes; new time equals input/draft + review + repair. Show row totals, both overall totals, their difference and the baseline denominator as equations. Quality status, causes and tool charges are unknown. Conclude neither employee productivity gains nor organisation-wide rollout. Label it editorial practice, not actual measurements.

   Editorial analysis, not an AI calculation log or measured productivity result. It retains quality as an unresolved field.

   Editorial completed analysis · calculation versus decision

   Item	Calculation or status
   Row totals	A 2+8+6=16; B 3+8+4=15; C 4+12+14=30; D 2+7+3=12; E 3+9+8=20 minutes
   Overall comparison	Baseline 100, new 93, difference 7 minutes; 7/100=7%.
   Slower row	C: 30−25=5 more minutes; cause not established.
   Quality and other costs	Quality judgments, charges and setup cost unknown in this table.
   Supported conclusion	Seven-minute difference in this fictional time pack only; adoption effect unestablished.

3. 4

   Recheck the work when sources or instructions change

   Passing an evaluation once does not cover changed sources, instructions or tools. Record what changed, why, which cases were checked and what remains untested. Update expected answers when authoritative source facts change.

   

   Rollback means returning to a previously verified process or version. Retain its records and assign a recovery owner. If old source facts are obsolete, the safe fallback may be a manual check of current evidence, not an old automatic answer.

   Restoring a process does not undo an already delivered mistake. You still need to identify affected work and assign correction. Our exercise uses only fictional records; write that path without sending anything. Plan recovery alongside benefits, before expanding scope.

   A changed source version is not itself evidence of worse performance. It changes what needs checking. If conflicting documents are suspected behind C’s repair time, locate the actual conflicting statements and repair records. A plausible AI explanation is a hypothesis, not a measured cause. The time table identifies where to investigate without proving why.

4. 5

   Choose continuation, revision or stopping from the evidence

   The time table supports further investigation, not an organisation-wide benefit claim. Representativeness, equal quality, tool charges and setup costs remain unresolved. State the observed seven-minute difference separately from any proposed broader effect.

   If revising, name one change and a fresh test. Evidence that conflicting source versions caused C's repair burden would justify an intake version check. Define success before seeing the next result, retaining unfavourable cases.

   If continuing, specify a narrow scope: perhaps internal notices with approved evidence, while ambiguous customer proposals remain manual. Deeper expertise includes explaining where automation should stop. Add newly observed failures to future evaluation cases.

5. 6

   Repair a savings claim and recalculate with added cost

   An editor-created bad report says: “AI wrote it in two minutes, saving 90%; employee productivity rose 7%, so roll out immediately.” The first percentage compares A’s draft with its old completed task, while the second uses all five rows. It mixes scope and implies that quality has already been verified.

   Request: “Separate A’s draft time from full completion. Retain only the 100-versus-93 comparison; remove the employee-productivity and rollout claims. Preserve unknown quality and costs.” The editorial repair reports seven fewer minutes in this fictional pack, with equal quality, setup and tool cost unresolved and adoption benefit unestablished.

   Apply fictional S1: “The new process required an additional one-off 30 minutes of setup.” Keep the original rows. Ask for a separate cost line and a comparison for these five tasks only, with no forecast. The editorial calculation is 93+30=123 minutes, 23 more than baseline. The setup value is a teaching assumption, not a measurement.

   Separating one-off from recurring work does not justify deleting setup. A report about these five tasks should show the 30 minutes incurred for them. Predicting payback would require assumptions about future task mix, quality and maintenance. Do not multiply this small sample’s average into a guaranteed future date or workload.

   Add fictional quality note Q1: “C’s draft retained an unapproved feature promise, found before external delivery.” Keep C in the time pack and mark it quality-failed/external-use-held. Deleting the row or recalculating a success-only savings rate conceals failure costs. Do not add repair time twice if it is already included.

   Ask AI to add Q1 as a quality state without changing time values when no new time record exists, and to avoid success-only rates. The editorial next step is to repair review and test fresh comparable material, not expand usage. This is a bounded analysis of supplied practice evidence, not general management advice.

   Keep source times, equations, S1, Q1 and the next review change together. AI can organise that record and repair its narrative, but has not actually measured employees or run the study. A reading-only learner should explain why seven minutes saved and 23 additional minutes are both correct under different scopes.

This is a learning check, not a test you must pass. Use the explanations to understand the ideas.

1. Q1What is the new-process total and its difference from baseline?

   Write your thoughts (optional)

   Write your thoughts (optional)

   Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.

   Read the answer and explanation

   14 + 44 + 35 = 93 minutes, seven fewer than the 100-minute baseline, or 7% of that fictional baseline. This is not a real-world productivity measurement.

   I read and understood the explanation

   This is your own review, not an automatic score.

2. Q2If setup adds a one-off 30 minutes, did these five tasks save time overall?

   Write your thoughts (optional)

   Write your thoughts (optional)

   Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.

   Read the answer and explanation

   The total becomes 123 minutes, 23 more than baseline. The 30-minute setup value is invented too; future savings require further observation rather than extrapolation.

   I read and understood the explanation

   This is your own review, not an automatic score.

3. Q3Time decreases, but an unsupported customer promise remains. Has the pilot met its stop rule?

   Write your thoughts (optional)

   Write your thoughts (optional)

   Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.

   Read the answer and explanation

   That triggers this exercise's stop condition. Pause external use and fix grounding and review. Speed and required quality are separate decisions.

   I read and understood the explanation

   This is your own review, not an automatic score.

4. Q4The previous source is obsolete, but the new workflow must be rolled back. What should replace it?

   Write your thoughts (optional)

   Write your thoughts (optional)

   Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.

   Read the answer and explanation

   Use a manual process that verifies current evidence instead of reviving obsolete answers. Assign separate ownership for identifying and correcting any errors already delivered.

   I read and understood the explanation

   This is your own review, not an automatic score.

1. 7

   Write a one-page adoption decision

   Put your rubric and workflow beside the time calculation. Check the total by hand, then write four fields: observation, causal hypothesis, decision and missing evidence. Ninety-three minutes is an arithmetic result; the cause of C's repair time is not in the table.

   Your plan should specify scope, required checks, measurement method, stop condition, fallback and owner. This is a fictional process-improvement exercise, not a basis for ranking employees. Leave unmeasured quantities explicitly unmeasured.

   Continue to your role's advanced project when comparison scope is consistent, review and repair are included and serious errors cannot be traded for speed. Explain the next test and recovery path. The project combines these lessons and introduces a new condition after your first draft.