实习日志 / English / Journey Research / A wrongly learned alphabet: our client manager did not understand real claims

A wrongly learned alphabet: our client manager did not understand real claims

中文 | English

Our claims process has a big problem right now. Specifically, the client manager we wanted has been built, but this client manager's knowledge is wrong. It's like someone taught from childhood that the alphabet goes qazwsxedcrfvtgbyhnujmikolp; that is wrong at the root. Our system isn't wrong to quite that absurd a degree, but it has no real understanding of what a person's first reaction is after an accident, where the pain points of traditional claims reporting are, and where the pain points of reporting itself are.

That is what I handed the Agent at the end of September. By then the claims agent could already chat smoothly, extract fields and judge the product family, and it looked convincing. What made me write those lines was the Sprint 4 presentation.

The Sprint 4 presentation

It was the last regular sprint presentation before we went to the AWS office. After watching the demo, our supervisor pointed out that the claims process had to match how claims really work: however complete the interface, if the business logic is not real, the product cannot stand up.

The supervisor gave a few examples from real claims. When a car is rear-ended, in reality you first record the other party's plate, driver and insurer; when someone is hurt, the police usually get involved and there is a police report; and who is at fault decides whether the customer pays the excess. Our demo's forms and flow had none of these. Showing only "accident, then submit the claim" skipped steps every real claim goes through, so the scenario itself did not hold up.

The supervisor suggested we do more research quickly, asking insurance practitioners and people who had actually made claims: what is really asked during a claim, which information is mandatory, what can be replaced by police documents, and how things differ across policies and liability. First understand how the claims process actually works; otherwise all we build is an interface that looks like claims.

Digging out the original research

Early in the project we had in fact done research, based on the published claims processes of State, Tower, NZI and Westpac, and built a baseline of how reporting works today:

Incident or loss occurs
→ ensure safety and collect initial facts
→ choose a reporting channel
→ verify the reporter, customer and policy
→ determine the claim type and enter the matching path
→ provide core incident information
→ answer claim-specific or event-triggered questions (as needed)
→ upload photos and supporting documents (as needed)
→ confirm declarations and submit
→ create the claim and issue a claim number
→ completeness check and initial classification
→ triage: standard cases to human review / rule-eligible cases processed automatically / special cases to brokers, assessors, repairers or specialists

It also compiled a classification table of common reporting fields, from reporter, policy, incident, time, place and loss to vehicle, driver, third parties, witnesses, police, injury, road and weather, home, items, theft and declarations, each marked as a core field, a conditional field, or specific to one kind of claim.

Comparing that table with the system's field registry at the time showed the gaps at once:

The research also had a section on a common weakness of competitors: context lost when an Agent hands over to a person. It cited outside evidence, such as Salesforce research finding that 56% of customers often have to repeat themselves to different representatives, and split the continuity of handoff into three levels: can you reach a person at all; is the original session kept without re-verifying identity; and does the person receive the full conversation, structured fields, attachments, completed steps, missing items and the reason for handoff, so they can continue from where it stopped. Only the third level really avoids the weakness. The research was also clear about the limits of its evidence: this is a general industry phenomenon, but whether a particular competitor has the problem must be verified by direct testing or interviews, not assumed.

Measured against this, the pain points came in two layers. The first was the pain of traditional reporting, such as context lost on handoff, backed by outside literature in the research, though whether our own system had this problem had not been tested. The second was the pain of reporting itself, what a person's first reaction is after something happens and what state they are in, and neither document had an evidence base for it.

So the supervisor was right. Our system treated "smoothly finishing a conversation, extracting safety-related facts and judging the product family" as the end point of reporting, while a real claim needs these fields to support a chain of decisions: establishing liability, calculating the excess, contacting the other party's insurer, verifying police documents. The problem went beyond numbers; "a few missing fields" was only the surface. The root was a gap in our very model of "what a real claim must collect, and why".

Three questions dug deeper

On 28 September, building on that diagnosis, I asked three more specific questions, each to be answered against the code, not from impressions.

One: with the Agent as the single entry point, how should the three scenarios begin? At the time the product family was left entirely to the model to infer from text, and the entry was a plain chat input. The conclusion: three very light scenario hints could sit next to the input (vehicle accident, home damage, damaged belongings); clicking one puts matching text into the first sentence without blocking direct typing. It just gives the model an optional strong signal and does not change how the model judges the family itself.

Two: when the customer wants to claim, how are liability and loss assessment handled? Should the Agent give advice? The policy wording itself hard-codes the permission boundary: liability is a professional decision and is never made by the Agent; the Agent can only collect facts, identify missing details or evidence and quote clause text, while whether cover applies, liability, approval or decline, disputed excess and payment decisions must all go to staff. The boundary is restated repeatedly in the permission and safety sections of the prompt; it is a deliberately designed red line, not an omission. The only existing automatic signal fired when the policy lookup itself failed, and had nothing to do with judging the case. If the Agent were to give a preliminary suggestion such as "the other party looks more at fault, roughly ninety-ten" for staff to rule on, it would need an entirely new kind of proposal, reusing the four-state ruling flow staff already use for signals: confirm, dismiss, override, resolve. That would not cross the red line, since the decision stays with staff; it upgrades a "mute relay" into "a relay with a draft". But it would be new design, not an existing capability.

Three: how well is basic information covered? The six elements, time, place, people, cause, course and result, were all actually in the field registry; their required timing was just spread across different states, never asked together at the entry. The profile held only identity and contact details; vehicle make and model sat in "my assets", saved voluntarily by the customer and optionally carried over; the policy was stored only as an opaque policy number string. My conclusion at the time was that even if the customer had saved a policy number, the Agent could not work out the real excess, because the system had no idea which policy, tier or excess that number corresponded to; the blocker was a missing mapping table in the data model from policy number to specific terms and excess.

That conclusion was proven wrong the next day. More on that below.

What the right process should look like

On 29 September I asked the Agent to change direction: stop pushing on along "find the gaps", first think independently about what the right process looks like, then check it against outside sources. It first corrected two assumptions it had been led into:

A free-text entry is not wrong in itself. Industry training on claim calls actually stresses that a shaken reporter tells the story first, and good practice is to let them finish before filling in policy details, rather than blocking them at the start with a classification menu. So scenario hints can only be "an optional convenience", not a sign that "the entry design has problems and needs a big change".

We had actually got the timing of claim numbers right. The system assigns a number as soon as the conversation starts, before any field is collected, and what is finally submitted to the downstream claims service is the external claim. This matches industry practice: a claim number should be issued at first contact on minimal information, not after collection is complete. This item should not be on the problem list.

Then it wrote seven points of "how it should be" from the reporter's point of view:

1. Opening the page, I don't have to click a category menu first; I can immediately say what just happened, as if calling a real person, without being interrupted by structure;
2. From the first sentence, the system actively checks "are you safe right now, is anyone hurt"; real claim scripts put safety among the first few items, not recorded passively after the car model;
3. My account of what happened is read back for confirmation, such as "so you were rear-ended at such-and-such intersection, is that right?", instead of the next fill-in-the-blank question being thrown at me straight away;
4. Without asking, from the first sentence I know my report is in effect and has a number;
5. When I mention "the other driver was at fault too" or "someone saw it", I get follow-up questions: the other party's name, plate and insurer, the witness's contact details. This information is easiest to get right now and gone later;
6. When I ask "how much will I pay myself this time", I get a specific figure for this policy, not just a reading of generic clause text;
7. At the end I know clearly what happens next, who is responsible and roughly how long it takes; if I'm handed to a person, I know why.

From the client manager's side, a real client manager taking a call has an invisible priority list in mind but does not read questions off it: let the caller finish, tick off what is already there and what is missing while listening, and then fill the gaps with a few precise questions. Our mechanism of asking follow-ups by priority was heading the right way; what was missing was that the safety check should come earlier and more actively. The real gap was in the client manager's professional output, in two layers: concrete figures for the customer (look up the policy and you know the excess, scope and tier), and a preliminary judgement for the claims handler (on handover, add a line like "it sounds like the other party is more at fault, but you'll need to confirm", giving a colleague a starting point instead of handing over a blank form).

The permission boundary stays closed. Not ruling on cover, liability, fraud and payment reflects real regulatory concern in insurance about AI making judgements; it is not a product failing. The right design has the Agent actively check safety, listen to the whole story before asking precise follow-ups, give policy-specific figures, form a verifiable preliminary judgement for a person, and explain the reason when handing over. Of those five, only "follow-ups by priority" was right at the time.

A judgement I got wrong

The same day, the teammate responsible for the backend submitted a field completion request, to grow the fields from 28 to 52 in an ongoing PR, covering third-party property, number of people injured, loss mitigation and policy linkage. I verified it item by item and wrote a confirmation and authorisation.

The verification was pinned to the two commits the teammate gave, not done from impressions: I checked out main and the PR's latest commit separately with git worktree and counted the field registry directly, 28 on main and 52 on the PR, a difference of 24, matching the request; I ran the test command given, 65 passed, matching too; I spot-checked two items listed as "already on main" and both were there as stated, nothing exaggerated. The request reported a problem where "providing only the other party's insurer returns 500", and on the PR I traced it to a root cause one level deeper than the request described: the request model's validator counted insurer name, policy number and claim number towards "at least one must be provided", while the stored record's validator recognised only name, phone, email and plate. The request was let through and the storage layer rejected it outright, hence the 500. It is a textbook case of request-layer and storage-layer validation drifting apart, and I suggested writing it as a regression test rather than just fixing it.

During verification I noticed that the policy fields listed in the request included excess, currency and cover items, which directly overturned a judgement I had written into an issue the day before. I had only looked at the bare policy number string in the account and concluded that the system had no idea what policy the number corresponded to. Checking the full chain showed that at reporting time, the system takes the policy number the customer confirmed and, through the policy lookup adapter, retrieves policy facts including excess, currency, cover items and effective dates. That was a real mechanism already connected end to end.

The real gap had to be restated in two layers: first, there was only a simulated adapter and no connection to a real insurer's system, which was never in scope this time; second, the policy facts held only one overall excess, while the clauses describe several additional excesses for driver, age, use and so on that may apply together, and whether that granularity was enough was exactly what the request was trying to solve.

In my reply I wrote a section titled "Something I need to correct myself on: my own earlier judgement was wrong", and then changed that issue's description from "the data model does not exist" to "simulated adapter plus insufficient granularity; align with this teammate's policy linkage work, without modelling it twice".

I also thought carefully about how to write the authorisation. Within a few confirmed boundaries (no liability decisions, excess rulings or final payments, no connection to real insurers' APIs, no change to the Agent's conversation strategy), field naming, enum values, record structure, fix order, whether to add fields or extend existing records, and whether to split into smaller PRs were all for this teammate to decide, without coming back to me for confirmation; the reply itself was the record of authorisation. The only case that needed syncing back was finding during implementation that the work had to go beyond the "out of scope this time" list, for example having to change the Agent's conversation strategy for some field to actually be used. The reason had nothing to do with permission: that work belonged to a few issues I owned, and I needed to know so the two sides did not duplicate it. Finally I wrote down what I would do on my side: change the description of my issue, and make no changes or comments on the teammate's PR and issue, since that was their work.

After two days

The supervisor's cold shower made us lay our system side by side with the real claims process. The conclusion went deeper than I expected: the problem was not a few missing fields but that the system's understanding of "what a claim must collect, and why" was wrong, like a wrongly learned alphabet.

Just as important in those two days was a correction of my own. My judgement that "the system cannot work out the excess" rested on an impression from looking at one place in the code; the teammate's request forced me to follow the full chain, and only then did I find the mechanism had long been connected. Verification has to be pinned to specific commits, conclusions have to survive a check along the full chain, and when you find you were wrong, say so directly in the document you write to the other person.