实习日志 / English / Agent Control / The wrong cut of meat: rewriting the agent from the user journey after fourteen rounds of patches

The wrong cut of meat: rewriting the agent from the user journey after fourteen rounds of patches

中文 | English

Say you are making yu xiang rou si, fish-fragrant shredded pork. We had spent all our time studying the seasoning, the side vegetables and the heat, but the dish needs pork shoulder blade flash-fried in oil, and we were using hind leg coated in starch. With the wrong cut of meat, no amount of care over seasoning or heat can save it. That was an analogy I gave the Agent in mid-September to describe our claims agent: everyone had put a lot of work into prompts, rules and fixes, yet the fundamental question, "what should the client manager do right now, and who decides", had always been handed to something that should not be making that decision.

This piece covers what happened around that analogy between 18 and 23 September. It started with a one-line issue, "agent orchestration", and ended with a PR that changed more than fourteen thousand lines, plus a short document that later became the judge for every change. In between was a long detour: fourteen rounds of patches, each one studying seasoning and heat, until I walked through it myself in a browser and confirmed that the meat was wrong.

Starting point: removing a drop-down

Next to the input on the claim page there was a claim type drop-down with motor, home and contents, plus Let Agent identify. We decided to remove it entirely: users should not have to understand Northwind's internal product categories before they can start describing an accident. The ability to recognise the product family from natural language stayed in the runtime; what went was making users choose it themselves.

Removing the drop-down pulled up the whole orchestration structure. I first wrote an orchestration consolidation plan built on a few boundaries:

• This turn's product family has exactly one authoritative conclusion. A new side-effect-free TurnFamilyResolver resolves it in the order "persisted family → confirmed fields → the single clear candidate in this turn → a constrained classification result → otherwise unresolved", and outputs the family, status (authoritative, inferred, ambiguous, conflicting), source and whether confirmation is needed. Routing, context, field contracts and third-party services all consume only this one result; none of them may work it out again with their own regex or old fields.
• The core reply and optional capabilities are separate. A turn has at least two kinds of outcome: the conversation outcome and the outcomes of any optional capabilities. If the core reply is invalid, the whole turn fails and state does not change. If the core reply is valid and one third-party candidate is not eligible, the chat is still saved and returned; that card just is not generated, and the rejection reason is recorded internally. An optional capability's failure must not be dressed up as "the model service could not complete the request".
• Copy and cards degrade together. Deleting only the failed card while keeping the model's "I've arranged an assessment for you" is not acceptable. Phrases such as "submitted, booked, contacted, done" must be tied to the runtime's real execution result.
• Third-party call failures go into their own lifecycle. An adapter timeout or exception goes into retryable, terminal failure or outcome-unknown, and must not escape as a failure of the whole chat endpoint. The split being removed was one where the message and reply had already been stored, a later adapter threw, the HTTP request failed, and the front end showed "not sent".
• Errors must be distinguishable. Model service unavailable, model service failed, the runtime's own contract error, optional capability rejected, third party unavailable and third-party outcome unknown each have their own code, and the runtime's own errors must not pose as provider failures.

Most of this plan's direction survived. But when the changes really started, a 503 turned up.

A 503, and an interrupted fix

I asked the Agent to look into the 503. The causal chain it found: if the user's first sentence contained a word like "assessment", the runtime judged the whole turn as "the user is looking for third-party support" and asked the model with a completely different question template, one with no fields for recording facts, so however much the user said that turn (rear-ended, location, whether anyone was hurt), none of it could be stored. On the second turn the runtime still did not know the family and went down another branch: no model call, just a hard-coded sentence stitched together, and that reply lacked the run record the audit required. Hence the 503.

The Agent proposed four local patches. I did not accept them. What I said was:

I gave you the documents and the codebase not to limit the direction of your thinking, only so you can see the defects clearly, and try to reproduce deeper logic holes through your own independent testing (for example in Chromium). Going further, although I can describe the final shape of the system I want in natural language, because the code has always been taken over by GPT, I don't know whether a robust user-journey state machine exists, or whether that state machine's design matches my design thinking. It's like making fish-fragrant shredded pork: we've been studying the seasoning and side vegetables, or the heat, all sorts of things, but actually the meat should be pork shoulder flash-fried in oil, not hind leg coated in starch. Do you understand? What I want is a teacher.

That had two demands. First, do not draw conclusions only from reading code; reproduce the deeper problems yourself in a browser. Second, what I really wanted to know was whether the system had a dependable framework of judgement representing "what the client manager should do right now", which mattered far more than fixing any one error.

The audit: the wrong cut of meat

To answer that, the Agent organised an adversarial audit with 29 sub-Agents, reading code and cross-checking, burning about 3.72 million tokens in total. Three diagnosticians with different viewpoints (product experience, formal state machine, the engineer taking over) each independently reached the same verdict: the state machine exists, and its shape is wrong. Of 8 key conclusions put through rebuttal, 6 were confirmed and none was overturned.

I broke the claims process I wanted into six points, and the audit compared each against what the code actually did:

The most painful one was injury. As soon as the user mentioned an injury, a regex matched and the claim was handed to a person at once, without even recording the boolean "someone was injured", and then the HTTP layer short-circuited: whatever the user sent afterwards got no reply at all until staff unlocked it in the workbench. A test even asserted this behaviour.

The audit also found the root causes of the 502 and the 503, two different illnesses at two providers. With Gemini, the output limit for the ordinary reporting profile was 180 tokens, while by the repository's own estimator a minimal valid response took 143 tokens, so a single field change was enough to exceed it; the three failed outputs were 163 to 165 tokens, cut off exactly halfway through the first field. With GPT, strict structured output had rewritten field changes so that "the parent object requires exactly 8 keys and each oneOf variant requires exactly 2 keys", which no object can satisfy, and the provider rejected it with a 400 before generating anything. The 503 came from a deterministic branch that produced an action code but no run record; the audit gate itself was right.

The audit report ended by coming back to my analogy: the meat was wrong, and it had nothing to do with the heat. The real "meat" of this system is one thing: who decides what to ask next and where to go, and on what basis. My answer was a client manager who can see the whole case file and the whole tree; the code's answer was a stateless function that recomputed the whole form from scratch every turn and picked the first unconfirmed field from a hard-coded list of 14. Beside it, the model only copied and made small talk.

But the audit was equally clear that the side vegetables were well cut and did not need throwing out. The fact ledger, where every field carries source, status, quote and an assertion chain, was the only single, explicit, enumerable state space in the system. "The model proposes, the runtime commits", optimistic locking and idempotent replay were all solid. Deriving required fields from confirmed facts, instead of walking the tree with a cursor, naturally let users speak out of order. Safety interception sat before the model, which is the right place. The third-party service registry, cards attached under the reply and consent recorded separately were all right too.

So what had to change was the meat, with three cuts made together: give the tree leaves (branch-specific fields, with branch activation changing the required set); give the model one unified channel for proposals (one turn can carry facts, the next question, service cards, state proposals and escalation together); and relax "what counts as remembered" (a quote found in the user's words is enough, the model's tidying no longer downgrades a fact, and only inferred, disputed or permission-affecting facts really need confirmation). Any one cut alone fails: leaves alone, and the model can still only copy; proposal rights alone, and it still faces a flat 22-field form.

I accepted this direction, opened issue #980 to track the rewrite, and folded #779, which handled recovery rules after model call failures, into the same PR. My decision was one complete implementation, no phasing.

Fourteen rounds of patches

What followed was a design spec revised from v2 to v3.15, fourteen rounds over about twenty-seven hours, all of it changes to the design document and not one line of code. Almost every round followed the same pattern: the implementing Agent (Codex) wrote code or ran acceptance against the spec, found a concrete contradiction and stopped to ask; the designing side checked, patched, and bumped the version. The Q&A numbering reached Q67.

Several times in those patches I turned the direction back on the spot.

Timeouts. One version set a 45-second timeout on model calls, after which the recovery path prompted the user to retry. I rejected it: "The agent's job is to handle the user's need and respond. At no point should a reply fail because of the model's own limits. What are you thinking? I'm now starting to suspect that the rules we set are actually holding the model back." Checking showed the provider calls were blocking, non-streaming requests that could not distinguish "the model is generating normally, just slowly" from "the provider really is down"; and 45 seconds had never been worked out from the worst case, only set from a feeling that "the experience should be smooth". The final rule tightened how long each field may be, recalculated the budget from the new lengths, and kept timeouts and recovery only for real provider failures.

Thinking mode. I asked whether the model's thinking could be turned off: the task itself only needed something at the level of GPT-4 or Claude 3, and we used newer models only because those were the earliest versions we could access. "Before, even a simple question took the model more than 80 seconds, which is very unreasonable." It turned out the runtime already had a switch to disable thinking, but it was wired only to the locally deployed model, not to either cloud model, so every call was waiting on invisible thinking tokens.

The family classifier was a regex. In testing I found that "I was rear-ended yesterday" did not load the vehicle branch on the first turn, and asked: "What do you mean it was missed? Is our unified family resolver a regex? That's absurd." The Agent's first fix was "a single candidate can light up the branch too", and I sent it back: "Does your fix solve the problem that the agent relies only on regexes and not on natural language understanding? I think downgrading the runtime's regexes to a suggestion layer is right, but let the agent judge the incident type through understanding (only the initial family sub-branch; our later logic is already in decent shape)... I want a solution that represents both our views, not a pile of technical description." That became the direction: while the family is unconfirmed, the specific fields of all three families are open to the model, which judges by understanding; regexes step back to being a reference for ordering follow-ups and a fallback for paths with no model.

The service catalogue. That in turn showed that third-party service recommendations depended on the same regexes. The implementing side proposed first showing the model a family-agnostic "candidate catalogue" and filtering again once the model had judged the family. I asked back: didn't we already stop the front end from forcing service pop-ups on users? Candidate services only need to hook into the backend's branch logic; it is a toolbox, you call what you need, with no need for a list driven by the dynamic form. On checking, this was more thorough than a "two-stage candidate catalogue": whether a service suggestion can become a card moved to the validation stage, looked up using the family actually adopted this turn, with no new intermediate state.

Several more were caught by the implementing side during acceptance, all quite representative:

• The user said "Yes we can still live here", which literally answered only "can the house be lived in", but the model set the "confirm the whole batch" boolean to true, so a family and damaged areas the user had never confirmed were confirmed along with it. The runtime computed the other four conditions itself from facts; only "should the confirmation really happen" was left to a field the model filled in.
• The model filled an address the user never mentioned with >UNKNOWN<, a non-empty string, so it was treated as "collected, pending confirmation" and pushed into batch confirmation; the user would be asked to confirm something they had never said.
• When the family was only "pending confirmation", the same field also counted as "missing", so the next turn asked about the family again, while batch confirmation required the family to be confirmed first. The two rules locked each other.
• When the user clearly said someone was injured and the model call timed out, the flow saved only the message and created no human handoff, because creating the handoff came "after the model succeeded". A safety signal the runtime had already detected was not acted on because the model failed.

Every one was a real problem, and every one was fixed.

Seven symptoms

As the fourteen rounds were wrapping up, I walked through the new version myself in a real browser and listed the symptoms for the Agent:

1. The first sentence already said the car was rear-ended, it was dark and the state of the car was uncertain, and the Agent still asked "can you describe what happened?";
2. The first sentence said the state of the car was uncertain, and the second reply asked about the same thing again;
3. The user replied "yeah", and the model invented "injury or danger has been recorded";
4. A car owner was asked "what was stolen?";
5. Internal field codes appeared in the conversation as-is, for example claim.product_family: motor;
6. After all eleven required fields were complete, an injury question popped up;
7. When a third-party service card should have appeared, it did not.

My last line was: "The whole flow is a mess. The current rules not only fail to help the model focus and work efficiently, they're making it talk nonsense." And I asked: "Why didn't you think about natural interaction from the start, instead of putting layer after layer of restrictions on the runtime?"

The designing side admitted it: from v2 to v3.15, every change was "the implementer reports error X, so the runtime blocks X", and not one section described what a good claim report looks like. All fourteen rounds improved "how the runtime verifies something", and not one asked "who should judge this". It had also pushed the principle "the model proposes, the runtime decides" from the fact layer into the conversation layer: choosing the model's next question for it, writing its summary for it, requiring a question every turn. That was ghostwriting, no longer "deciding".

Three things to get straight first

I did not ask for a new plan straight away. From an improvement plan alone I could not judge what the result would be after the change; I first needed to confirm that we understood a few basic concepts the same way. What I asked for was:

First give me, in plain everyday language, a complete restatement of the user journey, the state machine of the user journey and the dynamic form, and how each of the three relates to the runtime. Note: that's 3×2 = 6 things.

Once it had restated them and we were aligned, I added: "Write these six down first, in a journey-soul.md, and then you can diagnose the defects and prepare the rewrite plan. Remember, don't talk technology at me. But if in the end you can't implement it, don't give me something purely conceptual and too abstract either."

The file ended up just over a page long, and I restate its six points in full here.

The user journey

An experience from the user's point of view. Zhang San is rear-ended on Symonds Street in the evening, opens the app and talks to a client manager. He says what happened; the client manager notes it, asks a question or two, and gives him an assessment card. He says "it's dark, I don't know what the car looks like"; the client manager says "no problem, the assessor will tell us after looking at it, and if you get photos you can send them to me". He says "that's it"; the client manager reads back what will be submitted, he says "yes", and the claim goes to the claims team. Three days later he comes back with photos, and the client manager's first words are "got your photos of the car, I'll add them in". From start to finish he feels he is talking to someone who knows the business, never that he is filling in a form.

Files can appear at any time in this experience, not only photos: a police report, a repair quote from the garage, a medical certificate, receipts for stolen items. They arrive in three ways, and the client manager must handle all of them: the user sends one unprompted, even just dropping a file without a word, and the client manager must recognise what it is and which claim it belongs to; the client manager asks the user for it; or the client manager knows the file should come from a third party, requests it, and files it automatically when it arrives. Users should not be asked questions like "do you have a police report?" when they cannot know whether they should.

Relation to the runtime: the runtime is invisible to the user. Users should never see field codes, never get templated questions, never be asked the same thing twice. The only legitimate moment for the runtime to show itself is when it blocks something for safety or permission reasons. Anywhere else the user can feel the runtime is a design failure.

The state machine of the user journey

The framework of judgement in the client manager's head: "what stage are we at, and what should I do this turn". It is not a flowchart of "ask this first, then that"; it is more like a response framework. On every sentence or file, the client manager judges three things: what this tells me, what stage we are in, and the single most important action for this turn. There are only a few actions: note it and ask one thing along the way, clear up a contradiction, hand over a contact card, ask the user for a file, request a file from a third party on the user's behalf, summarise for the user to review, propose submitting, hand over to a person, or simply answer the user's question. The stages are roughly: understand what happened, fill in what can be filled, arrange the outside help that is needed, summarise and submit, and after submission come back any time to add things. A first report, coming back three days later with photos, and staff sending a follow-up question back all use this same framework.

Relation to the runtime: the model decides "what to do"; the runtime decides "whether it can be done" and "how to make it happen". The runtime checks whether the action is eligible (the user says "submit": has the claim reached a stage where it can be submitted? the model says "I'll arrange a tow": did the user authorise it?), carries out the consequences of actions (really creating the claim, really dispatching, really creating the handoff), and holds the safety floor (the user says someone is hurt: whatever the model wants this turn, the runtime creates the handoff first). It does not generate conversation content and does not decide where the stage goes; choosing the model's next question, writing its summary, requiring a question every turn are all overstepping.

The dynamic form

The claim form finally handed to staff, except that it grows along with the conversation. At first it has only the few common fields every claim needs; the user says "my car was rear-ended" and the motor fields grow; "my passenger was hurt" and the injury fields grow. It only ever grows the fields this claim needs. Every field ends in one of four places: filled in; the user gave a rough answer that is good enough, without chasing precision; unknown now but known later, marked "to follow"; or something the user should never be asked, left for a third party to fill.

The form has file fields too, with the same four endings. When a file arrives it does more than fill one box: the client manager should see whether it can fill other boxes along the way, and whether it matches what the user said. A value read from a file is the client manager's interpretation and starts as "pending confirmation".

An "unknown" field is not a defect; it is legitimate content of the form. Staff seeing "Vehicle condition: awaiting assessor's report" know what to do next. Before submission a field can pass through empty, pending confirmation and disputed, and those intermediate states do not block submission either; they go across with their markers, by the standard of "staff can take it from here". Every field must be designed as a field on the form, never as an admission gate.

Relation to the runtime: the form grows in the runtime's hands, and the hand that fills it is the model. Branch rules are the runtime's registry, a skeleton fixed by product design; the model cannot invent a field. What goes in each field, who said it, whether there is a quote behind it, whether it is vague or unknown, is proposed by the model from the conversation, and the runtime verifies the source and commits it. The runtime presents the whole form to the model but does not decide from the form what to ask next; that belongs to the client manager looking at the form.

How to use the file

Put together: the conversation makes the form grow, the state of the form tells the client manager what stage we are at, the stage decides what to do this turn, and the action produces new conversation. The runtime holds all three up from underneath, guarding the skeleton and the facts, carrying out irreversible actions and making sure everything survives an interruption, but it never speaks and never makes the client manager's decisions.

The file ends with four test questions to apply to any existing or proposed rule:

• Does it make the user feel the runtime? That violates the first point.
• Does it decide "what to do this turn" for the model? That violates the second.
• Does it treat "unknown" as a defect, or design a field as a gate instead of a field on the form? That violates the third.
• Does it make the user guess "should I have this file?", or leave an unprompted file unrecognised? That violates the first and third.

Only a rule that passes all four is legitimate. The file was extended twice more at my request: drawn as a flowchart with arrows, marking what drives each branch; and treating files as content on a par with conversation text across all three paths.

One more thing happened while the file was being written. A teammate raised 10 "contradictions", and for a while the Agent accepted all of them and started adding layers to the implementation. I corrected it: "Follow the design principles strictly; this isn't patching." In the end 5 were rolled back, keeping only 3 that were genuinely drawn wrong in the diagram and 2 wording changes.

Rewriting by the six points

v4 changed the order of the structure. First came three target journey scripts (a motor rear-end, a burst pipe at home, and a theft where the user starts by dropping a police report and nothing else) as the judges. Then a charter (who decides the model's nine kinds of action, the four things the runtime owns, and an explicit list of what it does not do). Then v3.15's fifteen invariants and every chapter were sorted one by one into "keep, change, demote to observation, delete". Only then came the form, the per-turn flow, submission, build order and acceptance. Acceptance asserted the eight turns of the Symonds Street conversation one by one.

A few concrete changes:

"Cannot be filled" finally had a way to be expressed. The domain model already defined seven field states: filled, suspected, confirmed, disputed, said they don't know, waiting for a third party, replaced. The conversation logic layer had squeezed these into three, "filled, not filled, good enough", and squeezed out the legitimate state "said they don't know", so the model was forced to make up answers to fill boxes; that is where >UNKNOWN< came from. v4 restored the seven states and gave the model's proposals a disposition: value (a definite answer), claimant_will_supply (the user will provide it later) and third_party_pending (a third party should fill it). My constraint was "modify or extend the current codebase, don't invent your own fields", so no new enum was created and only existing states were reused.

The runtime stops talking. The old code had several places where the runtime stitched together its own words for the user: replacing a failed follow-up with a fixed English sentence, replacing the whole reply when a ruling conflicted with the model, inserting a fixed closing line when questions exceeded the budget. All of these were removed and the wording handed back to the model, with an automated check confirming that in the normal flow the runtime generates no text the user can see.

What is said must match what is recorded. With all the wording handed to the model, there had to be protection against its words drifting from the ledger, for example the runtime having just rejected a service while the model says "I've arranged a tow for you". A new check looks at whether the reply mentions a service that was just rejected or claims that something unauthorised has been done; if so, the runtime's ruling is sent back and the model rewrites once. This "corrective re-call" replaced the old practice of stripping whole sentences.

Submission means "staff can take it from here". The only thing that blocks submission is "an empty field with no ending"; fields pending confirmation or disputed can be submitted with the claim, carrying their markers. Anything added after submission is a supplementary record attached to the submitted claim and does not rewrite what was submitted. Users do not need to know the difference; staff do.

The design document says only how things should be done now. This was a writing rule I gave during the rewrite: "Stop adding process content to the design document." From then on, "what v2.3 found and who rejected what" all moved into a separate changelog, and the design document itself states only the current rules. Implementers do not need to read the changelog.

Once the six-point file was settled, it became the judge for every later code change. Even comments in the code would say "this follows the journey-soul design" to show that a behaviour was intentional, not an omission.

Merge and review

Real coding started after I told the designing side "this time you're doing it yourself". When it was time to merge back into main, several teammates had pushed three partly overlapping features on main in parallel: the payout contact flow, a fix for duplicate message submission, and duplicate protection on claim creation.

The most important judgement in the merge was whether to keep talking to the user during a human handoff. Our design kept calling the model as normal during a handoff and only stopped recommending third-party services; a code comment said "a handoff changes the owner, but shouldn't make the conversation go mute". A teammate's independently built design on main did the opposite: during a handoff, new messages from the user are only recorded, with no model call and no reply. At first I thought a handoff meant only "the claim has been created and staff have taken over"; once I checked, its scope was wider: a user asking for a person, accessibility needs, emotional distress and urgent safety risks all counted. With the scope clear, I decided to follow main's silent design completely. A few hours later another boundary appeared: if a message had not gone through earlier because of a network problem and the user resent it, did it count as a new message during the handoff? My judgement was that silence blocks only genuinely new content, and retrying an interrupted old request goes through as normal.

The automatic merge also hid two kinds of problem, both "no visible conflict, but the wrong side chosen": a front-end feature was silently dropped without a conflict being reported; and an old handling logic that should have been retired was revived as a duplicate definition with the same name, so at runtime the later-loaded old version would quietly overwrite the new one. Both were found by comparing code line by line.

Once CI was mostly green, a formal code review report arrived listing five blocking problems. My instruction was: "Verify first and locate the core problem before concluding. Then attach the fix. Only check, don't touch the code." Only after they were confirmed did I authorise fixes, with "under no circumstances break the message handling logic that already works". Three of the five were the same principle not fully applied across different code paths:

• When the model marked "the user will provide it later" with an empty value, the runtime did no quote verification at all on this path, so the model could claim out of nothing that "the user said they don't know". "A label cannot replace verification" had already been fixed on another path; fixing one place does not make the whole principle hold.
• The reply check had an exemption allowing the model to say "I'll submit the claim to the claims team", since that is its own job. But the exemption was too broad: "I will submit" and "I have submitted" both passed, so the model really could lie that it had submitted. The fix separated future from perfect tense and exempted only the former.
• For a "third party should fill this" label, the runtime only checked whether the service could be proposed this turn, not whether the user had really agreed to its involvement.

The other two: reopening a claim pushed all the evidence files the user had sent before into the current input box; and a regression we introduced ourselves in the merge, where replacing a whole block with main's version deleted a small piece of the old version that distinguished "this is an error record" from "this is a successful response", so a case that should have returned a conflict returned "not found". The lesson of that one was direct: replacing whole blocks in a merge conflict has a price, and without line-by-line comparison a piece of code written specifically for an edge case disappears silently.

What it cost

The PR added 14,482 lines and deleted 2,079, over about five days. The first audit burned about 3.72 million tokens; the design document was patched for fourteen rounds over about twenty-seven hours without touching code; after the patch route was judged a failure, the rewrite was landed in ten steps; the merge went through two rounds of conflict resolution, and an external review found five blocking problems; the full local test suite grew from just over 2,900 tests to 3,031. The development environment also switched models twice: once from a large model to a lighter cloud model, more than tripling throughput; and once, after finding that the lighter model brushed off several clearly decidable fields with "later", switching only the model size and not the provider, to confirm in one step that the problem really was model capability. In the end this PR closed #976 and #980 and dealt with part of #779.

Back to the fish-fragrant pork

What this left me with is an order for making judgements. Design decisions stacked one by one do not naturally form a good experience; each of the fourteen rounds of patches was reasonable, and stacked together they made the model talk nonsense. The patches all fixed "how the runtime verifies something", while what had really gone wrong was "who should judge this". So in the rewrite we first defined what a good claim report looks like, then worked backwards to the design decisions it needs, and from then on every design decision had to say which of the six points it serves. One that could not say had no reason to exist.