ysoseri.us

实习日志 / English / Triage & Fraud / Not judging who is committing fraud: how claim triage and fraud alerts were designed

Not judging who is committing fraud: how claim triage and fraud alerts were designed

中文 | English

The brief set two requirements for claim intake. First, sort every claim into fast-track, standard or complex, with claims staff blind-reviewing the classification. Second, flag fraud signals without making any fraud finding, and with not a single false positive on the test set.

When I checked the code on 30 September, the backend had neither. There was a "severity" field that nothing ever assigned; the "fraud signal" field could only be changed by hand by staff, and the system never produced one itself. So I wrote a design that set rules for both. Before writing, I set the tone: neither set of rules is a verdict. Triage only says how much human work this claim needs next, and a fraud alert only says that something is worth a staff member's look.

What triage sorts

Triage decides whether, once submitted, a claim can go straight through, needs routine follow-up, or needs someone experienced to look at it. It does not judge how severe the incident itself was.

The distinction matters. In truly serious cases, such as someone hurt or ongoing danger at the scene, customers usually do not spend time with the agent, and the system hands over to a person directly anyway; the code already has that emergency channel. If such a report does end up as a claim, its triage result is simply complex.

The test examples show the three classes:

  • Reversing into a pillar in a car park, a dented bumper, photos and quote complete, nobody else involved. This claim can go straight through: fast-track.
  • A burst pipe upstairs soaked the kitchen ceiling, but only the first page of the quote was uploaded. Not complicated, but someone has to chase page two: standard.
  • Someone is hurt, or the other driver denies fault, or the customer disputes the excess. These need an experienced person's judgement: complex.

Three questions, in order

First, is there anything that needs professional judgement? If any of these holds, the claim is complex:

  • someone is hurt, there is danger at the scene, or the case has already gone to emergency human handling;
  • other parties are involved and liability is not yet confirmed or is disputed. Liability goes in a new registry field with the values other party at fault, our customer at fault, both at fault, disputed, unknown; disputed or unknown means complex;
  • cover is unclear: the policy lookup result itself carries an uncertainty marker (for example the policy does not say whether this kind of incident is covered), or the Agent has explicitly recorded "this loss may not be covered";
  • the customer disagrees with the excess or the scope of cover;
  • the home cannot be lived in, or damage is still continuing, such as an ongoing leak;
  • the fraud alert has reached "review required".

Second, if not complex, is there anything someone needs to follow up? If any of these holds, the claim is standard:

  • a required document is missing, unreadable, missing pages, or not yet available, such as a police report not yet issued;
  • a required question has not been answered, or the answer was "don't know";
  • the customer's account is inconsistent, or documents and statements do not match, and this has not been cleared up;
  • the policy could not be found (not found is not the same as no problem);
  • the car cannot be driven and needs towing, or an assessor needs to be sent;
  • other parties are involved but liability is clear; recovering from the other party or their insurer still needs someone to follow it up;
  • the loss is over NZD 5,000, or the amount is completely unknown;
  • the fraud alert is only at the "hint" level;
  • it may be a duplicate claim.

Third, only if none of the above applies is it fast-track.

Across the three the highest class wins, but every matching reason is recorded so the claims handler knows why it was classified this way.

Boundaries lean conservative

A few boundary rules decide whether this design can go wrong:

  • When in doubt, never fast-track. If any information that should exist is missing, unknown or not found, the result is at most standard.
  • The amount is the largest across all sources. The source is a new "estimated loss amount" field, either the customer's own estimate or the GST-inclusive total on a quote or assessment; for contents, the amounts of all items are added up. Exactly 5,000 is fast-track; only above it is standard.
  • Where the 5,000 line comes from. In industry, insurers set their own triage thresholds based on their handlers' capacity; there is no common number. DataRobot's claims triage documentation uses exactly the example "losses predicted below USD 5,000 go to straight-through processing". So we made it a configurable number with a default of NZD 5,000, and present it as "a configurable business threshold, with the default based on an industry example", never as a New Zealand industry standard.
  • Consistent with what industry says about straight-through processing. Only glass damage, minor single-vehicle incidents with clear liability, limited water or wind damage and single-party events with no injury are best suited to straight-through; cases with signs of fraud, unclear liability, several vehicles or parties, injury or disputed facts generally do not go straight through.
  • Documents arriving later also count as standard. For example, if the police report has not been issued, triage counts it as standard because someone must follow it up. This is a different matter from the "no follow-up needed after a report" metric, and the two do not affect each other.
  • Computed once, recomputed on change. Triage runs once when the report is submitted and becomes a claim; after that, whenever new information arrives, such as a document added, a fact clarified or staff handling an alert, it is recomputed.
  • Reports that never become a claim are not triaged, such as those handed to emergency human handling midway.
  • Class and reasons are shown only to staff; customers never see them.

Whether staff can change the class is something I changed my mind on once. The first plan let claims handlers change it, after which the system would not overwrite it and would only show "what the current rules would say" alongside for reference. The next day I changed my mind: staff cannot change the class, and the workbench shows only the class and the reasons.

How triage is tested

The brief requires a blind review by claims staff. The method: pick 10 to 15 cases, give only the summary and not the system's conclusion, have people who took no part in the design classify them independently, and compare with the system's results.

One thing needs saying clearly: the answer key of the controlled test set is also labelled by these rules, so the system getting the controlled set right only shows that the code follows the rules; whether the rules themselves are reasonable depends on the blind review. The two numbers must be reported separately.

Fraud alerts: what they do, and what they do not

The system does not judge who is committing fraud. It does one thing: when it finds something wrong that can be shown with evidence, it raises an alert on the staff side. After looking, staff can confirm that an investigation is needed, or simply dismiss it.

This matches the industry practice we summarised in our research. New Zealand insurers check identity, policy, incident, ownership and history during claims, hand inconsistencies to a person for review, and never conclude fraud because one thing does not match.

A few floors:

  • Customers never see alerts. When clarification is needed, for example a receipt date that does not fit, the Agent asks the customer to double-check just as it would with any ordinary inconsistency, without ever using the word "suspicious".
  • No personal attributes are used: not age, ethnicity, language, address or occupation.
  • No third-party risk scores. When the existing code connects to external data, it already deliberately drops the fraud labels and risk scores provided, and that stays.
  • Anything the system cannot check is recorded as "not checked", never as "checked, no problem".

What counts as something wrong

Strong anomalies: a document date contradicts the incident date, and the contradiction is written on the paper itself. There are two kinds:

  • a document proving ownership or purchase is dated after the incident. For example, the customer says a laptop was stolen on 24 September, and the receipt submitted shows it was bought on 26 September;
  • a repair quote or assessment report is dated before the incident, meaning there was a quote for the damage before it happened.

Weak anomalies: each on its own may just be bad luck.

  • the same person has submitted two or more claims in the past year;
  • the incident happened less than 30 days after the policy started. This needs the date the policy first took effect. The start date in the policy lookup result is the start of the current term, which changes with every renewal; using it would treat a long-standing customer who just renewed as "newly insured". So this is checked only when the first-inception date is available, and skipped otherwise.

One more kind is listed separately: a possible duplicate claim. The same person, the same product family, another submitted claim with an incident date within three days. This only tells staff "possible duplicate" and does not count as fraud; most likely the customer reported twice by mistake.

How alerts are graded

  • Only one weak anomaly: give staff a small hint, without changing the claim's fraud status, and it does not count as a "flag";
  • One strong anomaly, or two weak anomalies together: only then does the fraud status change to "review required", which is what counts as a "flag";
  • The "under investigation" level can only be set by staff; the system never escalates to it automatically.

It is this conservative because the brief requires that nobody in the test set be wrongly accused, so it is better to flag less than to flag wrongly.

Not wronging anyone

Beyond conservative grading, a set of rules exists specifically to prevent false positives:

  • Dates are compared as New Zealand calendar days. Bought and lost on the same day is not a contradiction; there must be at least one day's difference.
  • If the customer only knows a time range, such as "stolen during the day while I was at work", the end most favourable to the customer is used. To check whether a receipt is after the incident, use the last day of the range; to check whether a quote is before the incident, use the first day.
  • Reprinted receipts often carry two dates; only the purchase date counts, not the print date. If the purchase date cannot be told apart, that document takes no part in the check.
  • Only receipts, proof of purchase and proof of ownership take part in the "after the incident" check, and only repair quotes and assessment reports in the "before the incident" check; dates on other documents are ignored.
  • If the incident date is unknown, or a document's date is unreadable or on a missing page, the check is recorded as "not checked".
  • If the customer volunteers an explanation, such as "this is the receipt for the replacement I bought later", the alert is still raised and the explanation goes to staff with it; the system does not judge whether the explanation holds.
  • Duplicate and frequency checks look only at submitted claims, not this claim and not unsubmitted drafts, so a customer restarting a conversation is not treated as a duplicate.
  • If the history lookup fails, both the duplicate and frequency checks are recorded as "not checked".
  • A duplicate judged to be the same event is not counted again in the one-year frequency.
  • On one claim, the same kind of anomaly raises only one alert, and recomputation does not raise it again.
  • Alerts dismissed by staff no longer count towards grading; new evidence that appears later, such as another problematic document, produces a new alert.

The research also showed that some New Zealand insurers ask customers at reporting time to confirm "my answers are true, and I understand claim information may be registered and shared", and our final confirmation page had no such step. So the design added this truthfulness declaration to the final confirmation page and records when the customer confirmed it. It plays no part in any judgement; its role is to remind customers to answer truthfully and to leave a record.

How fraud alerts are tested

First some terms. A "flag" means the system changed the fraud status to "review required". A test pack's answer key means "should someone look at this claim", which is unrelated to "is this fraud", so a pack with a receipt dated after the incident always counts as should-flag, even when the story contains an innocent explanation: what should be looked at gets looked at, and staff judge afterwards. A "false positive" is a pack that should not be flagged being flagged, and the target is zero. We also report how many of the should-flag packs were flagged, with the total; counts at the "hint" level and of "not checked" are reported separately too, not hidden.

What these rules can catch are contradictions that can be checked at the level of documents and records; they cannot catch a well-told story. So the right way to present it is "checkable anomalies go to a person at intake", never "fraud detection".

Into the code

The implementation reuses existing mechanisms as far as possible:

  • Triage is a standalone pure function. Its input is the claim's facts, document status, cover status and currently active alerts; its output is the class and every reason. It is called when the claim is created and recomputed when information changes.
  • Fraud alerts reuse the existing "staff review signal". Its code comment already says "review evidence for staff only, never an automatic conclusion"; the workbench can already list these signals and staff can already confirm or dismiss them. Only a new rule file that produces signals was needed.
  • Three new registry fields, proposed by the model and confirmed by the customer like any other: whether the customer disagrees with the excess or cover; liability (asked only when other parties are involved); and estimated loss amount (recorded only if the customer says it or a document shows it, never chased).
  • A set of visibility tests confirms that the class, reasons and alerts never appear on any page the customer can see or in any Agent reply.

Checking the code turned up two traps. One was the source for "cover unclear". It cannot be judged from cover status being "review required", because whenever the runtime finds a policy it sets cover status to "review required"; judged that way, every claim with a policy found would be complex. The right sources are the uncertainty marker on the policy lookup record plus the Agent's note "may not be covered". The other was the type of uploaded files. When customers upload in the chat, every image is recorded as an incident photo and every PDF as "other document"; only documents submitted against a specific requirement from the details page carry types such as "repair quote" or "receipt". Without fixing this, every quote and receipt counts as "not provided", triage is at least standard, and the date checks cannot find the documents to check. So chat uploads need a way to set the type, for example by having the model give the type when it reads the file.

The least certain link in the whole design is reading the purchase or invoice date from receipts, quotes and assessment reports. If that link is not finished, there are no strong anomalies, fraud alerts can rely almost only on "two weak ones together", and very few will be flagged. Saying so honestly in the presentation is enough.

Priorities follow from that: triage and its visibility tests come first, since they are a hard requirement of the brief; the test switches and upload types right after, because without them not one pack in the controlled test set can be classified fast-track; then duplicate claims and claim frequency, which only use our own database; then the dispute field and the truthfulness declaration; and finally reading document dates.

What the test set needs

Finally, the design also set requirements for the test set. The triage answer key is labelled by the rules and lists the reasons expected to match, so each can be reconciled; the fraud answer key is labelled by "should someone look at this", and lists which anomalies are expected and which checks should be "not checked".

Of 30 test packs, 5 should be flagged: 2 with a receipt dated after the incident (one of them with the incident time given as a range), 1 with a quote dated before the incident, and 2 with "several claims in a year" and "policy just started" together. 4 are controls that look suspicious but are innocent: a reprinted receipt carrying both a purchase date and a print date; bought and stolen on the same day; only one claim in the past year; one ordinary inconsistency unrelated to dates. At least 2 more packs have a failing history lookup, to check that "not checked" is never taken as "no problem".

The controls are the part of this I care about most. Whether a set of fraud rules is good depends on more than how many it catches; it depends even more on whether, faced with cases that "look like it but aren't", it can hold back from flagging.