实习日志 / English / Policy Knowledge / When the policy has no answer: a rental car question that led to rebuilding the knowledge base

When the policy has no answer: a rental car question that led to rebuilding the knowledge base

中文 | English

In early October, while capturing footage for the demo, a very ordinary question got the Agent stuck: "Will my insurance pay for a rental car while my car is being repaired?"

At first it found nothing, then answered: "You could ask your insurance company." But Northwind is the insurance company. Worse, it claimed it had "escalated this", when no escalation happened in that turn at all.

The cause was quick to find: the motor policy simply had no rental car clause, so finding nothing was the truth. But how to answer after "nothing found", and why such a common question had no clause, pulled up problems with the whole policy knowledge base. This piece covers the research and design around it from 2 to 4 October.

How the market writes rental car cover

Northwind's policies are synthetic text for the demo, and the clauses only need to be reasonable and workable, but I did not want to make them up from nothing, so I first looked at how three New Zealand insurers do it:

Four shared points are clear: the claim must be accepted; the vehicle must be unusable; there is a daily or total limit; and it stops once the car is repaired, recovered or the claim is settled. The original PDFs from AMI and Tower would not open at the time, so the figures come from search result excerpts; that is enough for writing a synthetic clause, but not for citing as a real product.

Following those points, I wrote a clause MTR-BEN-03 for Northwind's motor policy:

## MTR-BEN-03 - Hire car, rental car and alternative transport while your car is repaired

If Northwind accepts a claim for loss to the insured vehicle under MTR-COV-01, including theft, and
the vehicle cannot be driven legally or safely, is being repaired by a Northwind-approved repairer,
or has been stolen and not recovered, Northwind may arrange a hire car of the same or a smaller
class, or reimburse reasonable costs of taxis, ride-share or public transport.

- Limit: up to NZD 60 a day, for up to 21 days per event.
- Starts: when the claim is accepted; for theft, when the theft is reported to the Police and to
  Northwind.
- Ends: when repairs are finished, when the vehicle is recovered and can be driven, or 7 days after
  Northwind offers a total-loss settlement, whichever is first.
- Not included: fuel, tolls, parking, fines, and the hire company's excess or damage charges; any
  period while the vehicle can still be driven legally and safely.
- No excess applies to this benefit. It applies whoever was at fault; Northwind may recover the cost
  from a responsible third party.
- Agree the hire with Northwind first. Costs paid by the policyholder are reimbursed up to the same
  daily limit.
- This benefit applies only when the policy schedule lists alternative transport.

The last line is there on purpose. Without it, every motor policy would automatically include this benefit, and the case "this policy doesn't have it" would never exist, yet that is exactly the case customers most need explained clearly.

What answer the customer should get

Before touching the knowledge base, I wrote the goal as three conversations, all on motor:

• The policy includes it. The customer asks whether a rental car is covered during repairs, and the Agent answers: once the claim is accepted, alternative transport is up to NZD 60 a day for up to 21 days, your policy includes this, citing MTR-BEN-03.
• The part the cover excludes. The customer then asks whether fuel for the rental is covered, and the Agent answers: fuel, tolls and parking are not included, again citing MTR-BEN-03.
• This policy does not have it. The same rental question, asked on a policy whose schedule does not list alternative transport, and the Agent answers: the clause applies only when the schedule lists this benefit, and this policy's schedule does not, still citing MTR-BEN-03.

Two more rules apply to every answer:

• No decisions. The Agent never says a particular claim "is covered" or "will be paid"; clauses plus the schedule can only lead to "your policy includes this benefit, subject to the claim being accepted". Anything that needs a decision goes onto the waiting list for staff.
• No invented citations. A question no clause can answer must never get a clause number attached. Either the policy has a clause that explicitly says "not included", or the Agent says it cannot find the answer and puts the question on the waiting list.

More than one problem

Checking the code against those three conversations found six problems.

P1: the policies did not cover what customers commonly ask. Three policies, 15 to 16 clauses each, about 2,000 tokens per family. Motor had no rental car clause, and the version still carried an MVP prefix.

P2: retrieval matched words, not meaning. Reporting turns used a keyword ranker to find clauses, with the customer's own words as the query. It dropped words of two letters or fewer, a list of stop words (policy, claim, what, how, much...), and product nouns (car, vehicle, motor, home, house...); for word forms it only stripped a trailing s, so repaired could not match repair, rental could not match rent, and hired could not match hire; and scoring was "title hits times 4 plus body hits". The result: a clause was found only if it happened to repeat the customer's exact word. "Will my policy pay for a rental car while mine is repaired?" was filtered down to will, pay, rental, mine, repaired, and a clause titled "Alternative transport" scored zero. "Is my car covered?" was filtered down to covered alone. To make it work, clause text would have to chase every synonym and word form.

P3: at most 3 results, each cut to about 600 characters. The resolution budget for policy references capped at 600 tokens, set once in the planner and once in the tool schema; each clause chunk was cut to max_tokens * 3 // count characters, which with 3 results is about 600 characters each. But a complete benefit clause runs 900 to 1,200 characters, and MTR-BEN-03 above is about 1,300. The model never saw the end of a clause, and exclusions and "subject to the schedule" are exactly what is written at the end.

P4: the local environment was pinned to the old version. The code chose the knowledge base version in this order: the control plane's current release; failing that, the published record in knowledge management; failing that, the constant 'MVP-2026.1'. An ordinary local environment started with start-dev.ps1 had neither of the first two, so it always read the constant. Even if a new version was ingested, it would never be read.

P5: the Agent could not tell whether "this policy" included a benefit. The schedule field did exist: coverage_sections is a free list of strings, looked up by the policy number in the claim. But local and test environments both used a simulated policy lookup adapter whose schedules were a dictionary hard-coded in the code, with only two policies. A tester typing any policy number got "no evidence" every time, so no turn ever saw a schedule that included or excluded a benefit.

P6: nothing measured "did the right clause make it into the answer". There were 6 evaluation cases, but they tested only the search API, and none entered through a claim conversation; a turn showing "completed" did not prove retrieval had run.

Let the model pick the clauses

For P2 and P3 I did not build vector retrieval, because the policies are small: about 50 clauses across three families, about 2,000 tokens per family. So the model can read the directory of clause titles for the family directly, say which clauses it needs, and the runtime fetches them in full. Matching the question to clauses becomes one of the model's own judgements in that turn, needing no other model and no vector index.

A lookup turn became five steps:

1. The directory goes into the context. When a lookup profile has been planned and the family is confirmed, the context gains a policy_clause_directory of [{clause_id, title}] for that family and the current version, about 16 entries and around 300 tokens. The directory comes from clause chunks already validated and ingested, so it can never list a clause that cannot be fetched.
2. The model names clauses. context.resolve gains an optional clause_ids parameter, at most 3, each of which must come from the directory.
3. The runtime returns them whole. Named clauses are fetched with the same scope filters as now (family, version, effective date, visibility, permission, insurer), each returned in full without truncation, together with the policy schedule; IDs not in the directory go into not_found, and another clause is never substituted.
4. The citation rule is unchanged. Only clauses the runtime actually fetched this turn can be cited; the model cannot cite a clause the runtime did not give it.
5. No clause named, no clause given. When clause_ids is empty, only the schedule and guidance_status: no_clause_selected are returned, and the model either answers from the schedule or puts the question on the waiting list, without citing any clause.

The budget was worked out too: 3 full clauses of up to 1,500 characters are about 1,100 tokens, inside the new 1,200 cap; the directory is about 300 tokens; and in the local real-model runs of 1 to 3 October, lookup turns were only 14 of 149 turns, so the rest are entirely unaffected.

The old keyword ranker was not deleted. The internal search API, admin evaluations and other callers still use it; what changed is only that reporting turns no longer use it to find clauses.

Failure paths were each written out: if the knowledge base is unavailable, return the existing "unavailable" result and its limits, still return the schedule, keep confirmed facts, and have the model explain the limit or use the waiting list; if a family has no approved version, the same; if the family is not yet confirmed, there is no directory and no lookup; if the model names an ID not in the directory, only that one goes into not_found; if it names more than 3, the tool schema rejects the call outright.

Read the version from the manifest, not a constant

The fix for P4 used a function already in the repository that reads "the currently approved version for this family" from the ingestion manifest. The staff Agent had been using it for a long time; the two constants in the claimant Agent just needed replacing with it. The function returns nothing when a family has several approved versions, so the old MVP entries had to be marked withdrawn in the same change, or neither Agent would get a version. When it returns nothing, the answer is "unavailable", and it never falls back to a constant again.

New content and version 2026.2

P1 started with an inventory: one table per family, "how a customer would ask → which clause answers it → covered, partly covered or missing". Questions came from customer messages in the controlled test corpus, the demo scenarios and a starter list:

Every "missing" row gets a clause; for cover Northwind deliberately does not provide, a clause is written that says explicitly "not included".

Clause writing has a few rules:

1. Numbers only go forward, never renumbered or reused. Clause chunk IDs, tests and stored citations all depend on them.
2. Titles say plainly what the clause covers. The model picks clauses from titles alone, so it must read Hire car, rental car and alternative transport while your car is repaired; Alternative transport alone is not enough. That also removes any need for a synonym table.
3. Every part is spelled out: trigger, what is provided, limits (per day, total, duration), when it starts and ends, exclusions, whether the excess applies, and what the schedule decides.
4. Each clause stays within about 1,500 characters, one clause per subject, so three always fit in one lookup.
5. Synthetic text follows New Zealand market practice without copying any insurer's wording, with sources recorded in the PR.

The new version is 2026.2. Document IDs drop the version suffix, so each version is a new version of the same document; the opening drops "for MVP demo" and keeps "synthetic, not a real insurance contract". Every script, corpus tool, configuration, document and test that hard-coded MVP-2026.1 was listed for changing together; only tests of "ingesting or protecting a version" itself may keep a fixed test version.

Let the schedule decide "included or not"

P5 needs no new field and no database seed: coverage_sections gets a value alternative_transport, meaning the policy includes MTR-BEN-03. The hard-coded policy dictionary in the code gets a few more policies: one motor policy with alternative transport, one without, and one each for home and contents, with numbers testers can type, such as NW-MTR-204518, written into the README. When a number matches none of them, the result is still "no evidence", and the Agent says it cannot find this policy's schedule instead of guessing.

Measure it

All of P6's measurements go through the normal claim message API, in an environment started with start-dev.ps1, with the configured model, never through the search API.

Clause selection accuracy: every question in the inventory maps to a scenario and an expected clause. From each turn's run record, read the clause IDs requested and returned and the clauses finally cited, and report per family the share of expected clauses fetched and cited, plus every miss and the directory titles the model saw at the time. A miss usually means some title does not say clearly what it covers, or no clause can answer the question at all, so it is fixed in the content.

Four end-to-end scenarios, 5 runs each: ask about a rental car when the policy includes it; ask the same when it does not; ask a question no clause can answer, which should go to the waiting list with no citation; and stop MinIO, which should explain the limit, keep confirmed facts and cite nothing.

Acceptance maps one to one to the problems above: no "missing" rows in the inventory; the right clause found by meaning, with an accuracy report; clauses over 600 characters returned whole in lookup turns; 2026.2 read in a local environment with no release and no release record; the schedule decides included or not; no invented citations, and an honest explanation when unavailable; ingestion works on both a fresh MinIO and one with the old version.

What was left out, and how it was handed over

A few things were explicitly out of scope: vector or embedding retrieval (only when the clause directory one day grows too big for the context); prompt changes (owned by another issue in progress; if measurement shows the model ignoring the directory or schedule, report it with the turn records); retrieval for the staff Agent; real insurers' clause wording; and a production policy system.

I wrote this design in English, together with a complete prompt for a coding Agent, and handed it to the teammate responsible for the backend to implement. The prompt's first line tells it to read the repository's rules and this design first, and to verify every stated fact in the code before changing anything.

Looking again at that opening line, "ask your insurance company", it exposed three layers of problem: no answer in the content, retrieval that could not find answers close in meaning, and no honest way out when nothing was found. Adding one rental car clause would make that one conversation look better, but the next question not written into a clause would fail the same way. So three things had to be fixed together: content that covers what customers really ask, a model that picks clauses by meaning and sees them in full, and "nothing found" as a normal path.