实习日志 / English / Evaluation / Designing an evaluation for a claims Agent

Designing an evaluation for a claims Agent

中文 | English

Scenario 8's acceptance requirements are very specific: 10 kinds of scenario (5 motor, 3 home, 2 contents); 80% of claims needing no further contact; severity triage blind-reviewed by staff; zero false positives for fraud alerts on the test set; and the whole report taking under 5 minutes and fewer than 10 questions.

These numbers only mean something inside an evaluation that holds up. What is being tested is an Agent that chats, and the same sentence gives a different result on every run; the customer it faces can only be played by another model. The evaluation first has to answer a few things: what kind of journeys are being tested; where the answer key comes from and how it must be written for a machine to compare against; how the model playing the customer can behave like a real person without doing the Agent's work for it; and how scores are calculated, and what the published numbers include and can show.

The evaluation has two parts: a controlled corpus that can be run again and again under identical conditions, and a test harness that runs it. The corpus and its standard document were made by a teammate; the requirements and review comments were mine.

A test pack

One pack is one complete claim journey; a single image or document on its own is not a pack. Each pack has four parts: the answer key, the customer's first message, every fact the customer knows, and the materials to upload. A separate setup.json says what state the harness must put the system in before the run.

CTL-M-001/
  input/
    initial_message.txt
    facts.json
    materials.json
    materials/
      photo-rear-bumper.jpg
      repair-quote.pdf
  oracle/
    labels.json
  setup.json
  provenance.json
  review.json

What models may see goes in input/, the answers in oracle/, in separate folders. input/materials.json lists only the file name, evidence type and when the customer has the file; each image's generation purpose, prompt and the fields printed on documents would all give the answer away, so they moved into provenance.json, which only the checker and the scorer read.

First messages come in six styles: plain, many_facts, emotional, typos, non_native and wrong_product_term (for example calling contents cover "house insurance"). The style changes only the wording, never a fact: "water came through the ceiling" may become "water started coming through my ceiling!!", but not "water was pouring". Dates are always absolute. Writing "last night" would mean a different day on every run, moving the incident date, and an innocent pack could run into a date check.

Answers derived from rules, able to catch false positives

When the first 3 sample packs (version 0.1) came in, my comments all concerned how the answer key was written, and they became the changes in version 0.2.

No innocent controls. With only packs that "should be flagged", false positives cannot be measured, and zero false positives is exactly what the brief requires. 0.2 added dedicated innocent controls: things that look suspicious but under the rules should not be flagged.

Fraud and conflict mixed together. The fraud label means "does a person need to look at this claim", which is not the same as "this is fraud", and it cannot be set automatically just because some boundary condition is present. In 0.2, labels come only from the rules' checks: a receipt dated after the incident and a repair quote dated before it are strong signals; two or more other claims in a year and an incident within 30 days of the policy starting are weak signals. Every check gets one outcome: hit, clear, not_checked or not_applicable. not_checked is never the same as clear: an unreadable date, a date on the missing page, a failed lookup, or data the system does not hold all count only as not checked, reported separately in scoring. Only one strong signal or two weak ones make review_required. A receipt dated after the incident is flagged even when the story gives an innocent explanation; the explanation goes to staff with the alert, and the system does not judge it on anyone's behalf.

The answer was a paragraph. A scorer cannot read prose. In 0.2 the oracle became structured: the triage class, the reasons that must appear, the reasons that may appear; one outcome per review check; and assertions with three kinds of judge, state for system state, text for wording, and human left to a person. Facts were split into two groups too: expected_facts are those the system must collect on this path, which must end confirmed with matching values; optional_facts are those a good Agent may also ask about, which must be right if asked.

A missing page could not be seen. A two-page quote with only the first page supplied gives no sign on the document itself that anything is missing. In 0.2 every page prints "Page X of N".

Images were only checked by AI. The photos in the corpus are generated, and a generated image does not necessarily show what the pack assumes. In 0.2, review.json records the assistant's check and a person's sign-off separately, and check_corpus.py --freeze does not pass until every image has a sign-off from a named person. 0.2 also added a blind look: send the image alone to the vision model the product itself uses, with no facts and no purpose, and record what it sees on its own. A close-up of a TV with a smashed screen was read as a glass coffee table, so that image was remade.

The model playing the customer must behave like a real person

The harness uses a model to play the customer and go through the whole journey with the Agent. The easiest thing to go wrong here is the test itself cheating: if the model playing the customer reads the answer key, it will "happen" to say exactly what the Agent needs, the journey will go suspiciously smoothly, the score will look good, and nothing will be proven. So what the customer can see has to be drawn clearly, by field rather than by file. The simulated customer can see:

• initial_message: sent as is, typos included;
• persona: personality and way of speaking;
• facts: everything the customer knows; anything not in it counts as unknown to the customer;
• the files in materials themselves, plus one plain-language label per file, such as "photo of the back of the car" or "the garage's quote". These labels are handwritten once per pack and cannot be converted from the system's type codes; a real person describing what they have never says incident_photo.

It must never see: the system's evidence type code evidence_kind, the prompt and purpose used to generate images, source images used only to make other files, setup, or anything in oracle.

What the customer can see and the labels live in separate files or objects, so that a code bug one day cannot slip the answers into the customer's prompt. There is also a test: if any value from oracle or setup, such as the expected triage class, appears in the customer model's prompt, the test fails.

The customer's behaviour follows a real person too: answer only from facts, and when asked about something not in the pack, say you don't know in keeping with the persona; do not volunteer what the Agent did not ask; do not correct the Agent, hint at what it should ask, or hand over files nobody asked for, unless the persona says it would. Materials the persona says will "come later" are described that way when first asked for, and uploaded at the time the pack describes.

Decide from the screen, record from the network

Reading the oracle is one way of peeking; another is more hidden: reading the data returned by the system's API. The responses contain much that a real person never sees: customer_next_step.required_items lists what is still missing, turn_record holds this turn's judgements, and the dynamic form's requirements and the current claim form are all in there. A harness that uses these to decide what to say next is no different from one reading the answers.

So every decision is based only on what is visible on screen: the text of the Agent's reply, the chips under it, the sidebar, the to-do bar, buttons and dialogs. The customer model receives only the visible text of the latest reply and of any card needing action, never JSON. All network data is recorded for evidence and scoring, but the customer model never touches it.

The environment has to be clean too. Before each pack runs, its own setup is loaded through a seed route, with a brand-new customer: a new anonymous session, or a new account if claimant_session requires one; anonymous_then_sign_in means start anonymously and log in when the product asks. Packs share no browser storage, accounts or claims. Runs happen in the local environment started by scripts/start-dev.ps1, with real MongoDB, MinIO and the configured model; if the backend reports itself as the controlled runtime, the keyword-replying stand-in, the harness refuses to start. None of this goes into CI; it runs by hand or on a schedule.

Scoring is after the fact, and mechanical

Scoring runs separately after the journey is over; only then is the oracle read, and comparison is mechanical, by rule. A model may help write a run summary, but it cannot decide the score. For each pack the record includes:

• triage: the class and reasons shown on the staff side, against the expected class, required reasons and allowed reasons;
• review signals: whether they fired, with false positives and false negatives counted separately and innocent controls reported on their own;
• facts: whether each key fact's value is right, with incident time counted separately;
• journey: whether a claim number was obtained, how many messages the customer sent, how many times the Agent repeated a question, how many replies left the customer with nothing to answer, how many retries, where it got stuck and why, and median and longest time per turn;
• staff side: how the customer's questions and contact requests along the way look on the staff waiting list.

Some capabilities were not finished in the system, such as review signals and the staff-side triage display at the time. When a pack runs into one of these, it is recorded as "not implemented in the system"; that is not a harness failure, it cannot be quietly skipped, and it is not counted in passes or failures. The published numbers include only what was actually measured.

Each pack keeps the full conversation the customer saw, a screenshot of every step, the network log, the customer model's prompts and replies, and the scoring sheet. Run output is stored outside the repository: run records are evidence, kept apart from fixtures.

What the score shows

Once the corpus is frozen as v1, answers and materials are never changed to make results look better; a real error leads to a new version, with the reason in the manifest's changelog. Failed runs stay in the results and are never deleted.

One more thing needs saying clearly: the controlled corpus is labelled by the same rules the system is meant to implement, so passing only shows that the code follows the rules. Whether the rules themselves are right has to be measured separately, by claims staff's blind review.

The order was set too: first the seed file format and seed route, the split of what the customer can see, and that leak test; then one motor pack run all the way from the first message to a claim number, plus reading the staff side; then 10 packs, with scoring sheets and each pack's time and cost; and once the 10 are stable, all 100. A mix for the 100 packs was also proposed: 50 motor, 30 home, 20 contents, of which 17 should be flagged and 13 are innocent controls.