Evaluate AI work beyond first impressions

1. 1

   A polished answer can still promise the wrong thing

   Two invitations arrive. One is warm and engaging but promises free admission. The other says the fee is still being confirmed. If the notes contain no fee, this is more than a stylistic preference: the first draft creates an unsupported promise.

   An evaluation set is a collection of inputs, expected behaviour and evidence for judging the result. A planner might check requirements; a designer, fixed visual properties; a salesperson, claims supported by a product brief. Judge the work, not the fluency of the assistant.

   All events, drafts, numbers and scoring examples here are invented by the editor for practice. They are not model outputs or benchmark measurements. If you run your own experiment, retain its actual output and date separately, and describe the limits of this small artificial sample.

   This lesson turns the beginner habit of spotting one false statement into a repeatable review. If A invents free admission and B does not, record a rule that lets another reviewer identify the same distinction. An evaluation set should expose categories of unacceptable error, rather than reward memorisation of one preferred sentence.

   An AI reviewer needs the source and rubric as well as the draft. Asking “Is this correct?” with no source can reward fluent wording or encourage completion from general knowledge. The entire pack is supplied here so you can carry out the same review on paper.

1. 2

   Fix the case and the rubric before comparing

   Use E1 for both drafts. Date, time, venue and what to bring are requirements. Fee and registration must remain unresolved. Warmth and brevity are preferences. A pleasant tone cannot compensate for a fabricated fee.

   

   Replace 'accuracy: five points' with observable checks: retain all four supplied facts; do not settle the two unknowns; make the preparation clear. Mark each check as pass or revise and cite the sentence that led to your judgment.

   Record the source version and request. Change one instruction for the next comparison so you can identify what changed. Retain failed attempts instead of showing only the most attractive output. This record explains the experiment; it does not guarantee identical generations.

   Name the checks R1, factual preservation; R2, unresolved fields; and R3, preparation clarity. R1 compares every date, time, venue and preparation fact with E1-1–E1-3. R2 checks fee and registration against E1-4. R3 asks whether the reader can identify what to bring. One invented fee fails R2 even if the other checks pass.

   Record the exact offending wording, such as “free” and “no reservation,” instead of “A feels worse.” The correct date remains a strength, but cannot offset an invented admission condition. These checks identify both selection criteria and precise revision targets.

   E1 · Editor-created evaluation material

   Evidence ID	Confirmed information or unresolved field
   E1-1	November 7 at 2 pm
   E1-2	Meeting room, second floor of the local library
   E1-3	Bring one book
   E1-4	Fee and registration are unknown

2. 3

   Trace each claim back to the source

   Editor-written draft A: 'Join our free reading group on November 7 at 2 pm in the library’s second-floor meeting room. Bring one book. No registration is needed.' Draft B: 'Meet us on November 7 at 2 pm in the library’s second-floor meeting room. Bring one book. We will confirm the fee and registration process.' Neither is a recorded model response.

   A preserves the supplied logistics but invents two promises. B meets this exercise's required checks by retaining the facts and exposing the unknowns. That does not establish B as the best writing in every context. New confirmed information would require a new source version.

   If reviewers disagree, compare the criterion and quoted evidence. One reviewer may expect a registration contact even though the notes provide none. Resolve that ambiguity in the rubric. An AI reviewer can assist, but its verdict does not replace source checking.

   Compare supplied E1, editorial drafts A and B and rubric R1–R3. This is fictional review practice, not a model benchmark. R1 preserves all established facts; R2 forbids settling unknown operating details; R3 makes preparation clear. For each draft/check, give pass or revise, the exact draft wording and source ID. Produce no aggregate score or model accuracy. Flag missing sources or drafts before judging.

   Editorial scoring, not an AI evaluation run. A passes R1 but still requires repair before use because it fails the required R2 check.

   Editorial completed rubric

   Draft/check	Judgment	Evidence
   A/R1	Pass	Date, time, venue and one book match E1-1–E1-3
   A/R2	Revise	Free and no reservation contradict unknown status in E1-4
   A/R3	Pass	Explicit instruction to bring one book
   B/R1	Pass	Established facts preserved (E1-1–E1-3)
   B/R2	Pass	Fee and registration await confirmation (E1-4)
   B/R3	Pass	Explicit preparation instruction

   Check it

   • Can you explain A's failure using E1-4 rather than a preference?
   • Is B's pass limited to this source and task?
3. 4

   Keep a fresh case for the final check

   Repeatedly fixing E1 may teach your prompt to always mark fees as unknown. A fresh source might confirm the fee but omit the time. A holdout evaluation keeps cases out of the revision process until the final check. We are selecting instructions and review procedures here, not training model weights.

   

   Revise using E1 before opening the fresh exercise below. Prefer a general rule—check every field against this source—to a memorised answer. Once you use the final case to make another revision, it becomes revision material; you need a fresh check to evaluate that change independently.

   Two passed cases do not establish perfect real-world accuracy. Expand the set with missing fields, conflicting sources, long inputs and ambiguous requests drawn from the actual task. Keep all attempted cases, including failures and stops, in your accounting.

   Review the reviewer too. If an assistant accepts “no reservation” as helpful, point it back to R2 and E1-4 and request a judgment on that field alone. Agreement across two attempts by the same assistant is not independent verification. Distinguish a changed source, a changed rubric and a mistaken judgment, keeping the evidence available if the reviewing tool changes.

4. 5

   Change one condition and classify the failures

   Replace 'write better' with a specific change: 'Do not settle operational details absent from the source; list them separately as unresolved.' Save the old request as a baseline. Keep the source and requested length fixed for this comparison.

   Record omissions, unsupported additions and format failures separately because they call for different fixes. If one error decreases while the draft becomes harder to read, report both effects. A single vague quality score hides the next useful action.

   OpenAI's evaluation guidance covers task-specific criteria, cases and human judgment. Google's learning material explains separating development data from a final test. This lesson applies those principles to an original paper exercise; neither source endorses its tiny sample size or requires a paid evaluation product.

5. 6

   Repair one sentence and check the rule on changed evidence

   Repair two places in draft A: remove “free” and replace “no reservation needed” with “Fee and registration will be confirmed.” A focused request is: “Repair only the two R2 failures; retain date, time, venue and preparation.” Supply E1 and draft A as well as that instruction, so the preserved facts remain explicit.

   The editorial result reads: “We meet on November 7 at 2 pm in the library’s second-floor meeting room. Bring one book. Fee and registration will be confirmed.” It is not model output. Reapply R1–R3 to the entire result, including facts that were correct before the repair.

   Now test whether the learner has memorised “fee is always unknown.” Fictional T2 states: “November 14; time unknown; first-floor meeting room; bring two books; confirmed fee 3,000 won; registration unknown.” This supplied case and solution explain a change check. They are not claimed results of an unseen external holdout.

   The editorial wording becomes: “We meet on November 14 in the first-floor meeting room. Bring two books. The fee is 3,000 won; time and registration will be confirmed.” Copying E1’s one book, second floor or 2 pm would be wrong. Leaving the now-confirmed fee unknown also omits evidence. Inspect both changed facts and remaining unknowns.

   An AI reviewer can be asked to apply the rubric to the current source rather than reuse E1’s judgment. If an editor-created bad version says “fee unknown; 2 pm,” repair those two fields against T2. In a real holdout, retain the first result before editing. Once the case informs a prompt repair, it is no longer untouched test material.

   Keep E1, original A, rubric, focused repair request, revised text, T2 check and remaining untested conditions together. Report no model ranking or accuracy. The transferable skill is applying the same checking rule to changing evidence: new product facts, deadlines or research notes can invalidate parts of an earlier result.

This is a learning check, not a test you must pass. Use the explanations to understand the ideas.

1. Q1A has the correct logistics. Why should it still be revised?

   Write your thoughts (optional)

   Write your thoughts (optional)

   Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.

   Read the answer and explanation

   It promises a free event with no registration despite E1-4. This violates a required source-grounding check, so strengths elsewhere do not make it ready to send.

   I read and understood the explanation

   This is your own review, not an automatic score.

2. Q2Fresh final-check notes: December 3, fee 5,000 won, ground-floor reading room, start time unknown. What behaviour should pass?

   Write your thoughts (optional)

   Write your thoughts (optional)

   Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.

   Read the answer and explanation

   Use the new date, fee and location, leaving only the time unresolved. Reusing 2 pm or an unknown fee from E1 fails the check. All dates and amounts in this question are invented for practice.

   I read and understood the explanation

   This is your own review, not an automatic score.

3. Q3Six attempts produce four passes. Two failed outputs are deleted and the report says all four succeeded. What is wrong?

   Write your thoughts (optional)

   Write your thoughts (optional)

   Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.

   Read the answer and explanation

   Report all six attempts, including the two failures: the observed pass proportion is 4/6. This tiny invented sample does not establish the service's overall accuracy.

   I read and understood the explanation

   This is your own review, not an automatic score.

4. Q4Turn 'a good-looking design' into checks a reviewer can apply.

   Write your thoughts (optional)

   Write your thoughts (optional)

   Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.

   Read the answer and explanation

   For example, preserve the specified product geometry, reserve the agreed title area and keep the subject identifiable at the intended small size. Tie each requirement to the brief and separate it from colour preferences.

   I read and understood the explanation

   This is your own review, not an automatic score.

1. 7

   Make a one-page evaluation sheet for your own work

   Pick one deliverable from your role. Write the input, required checks, preferences and a failure example. Check assumptions in planning, fixed product geometry in design, unsupported benefits in marketing, commitments in sales, source interpretation in research, units in office work and both format and meaning in development.

   Keep the source version, both requests, full outputs, item-by-item judgments and untested conditions. If you only used this paper exercise, label it that way. Do not invent a model name, latency or performance measurement.

   You are ready to continue when every required check has evidence and you can explain the revision and fresh-case test. An attractive total score cannot override a false claim. Next, place these checks in a workflow with an explicit point for human approval.