实习日志 / Internship Journal 1 & 2 (merged submission)

Internship Journal 1 & 2 (merged submission)

Project: Scenario 08, Northwind FNOL Agent, CS778 AWS Challenge Author: Leif (Zhixuan Wei) Period covered: Milestone 1 (user research and Prototype), Milestone 2 (MVP to date)

1. The project and how we work

Northwind is the fictional insurer in our scenario. It handles more than 80,000 motor, home and contents claims a year. According to the Scenario 8 brief, its current First Notice of Loss (FNOL) process takes 15 to 25 minutes, four in ten reports need at least one follow-up contact, and the average time from first report to claim creation is 2.5 days. It relies on two channels today, each with a cost: the phone line understands natural language and copes with exceptions, but staff time is expensive and hard to scale; the web form is structured and always available, but long and rigid, its questions never adjust to the incident, and someone who has just been in an accident has little patience for a long form. Neither channel collects photos and documents in any systematic way.

Scenario 8 asks us to build a claims Agent that can take a customer from their first sentence all the way to claim creation. We read this as a coordination problem: producing a polite reply is not the hard part. The hard part is turning an incomplete, sometimes jumpy account into a reliable claim record, judging what is still missing and what can already move forward, and handing the case to a person, with its full context, when professional judgement is needed. From this came our product position: let the customer tell what happened in their own words, let the system carry the insurance structure, evidence tracking and process behind it, and keep the claim moving to its next safe step. The position takes shape as two core capabilities: an adaptive intake, where how many questions get asked, and how deep they go, depends on the incident and the customer; and a single claim record that customer, Agent and staff all work on, so that context survives a handover intact and the customer never has to tell the story again to whoever takes over.

We are a team of five students. In this in-house internship project we work in two-week iterations, coordinating through a GitHub repository and a Kanban project board. My responsibilities in the project are: the Agent itself, its behaviour design and model integration; the design and rebuild of the claimant interface and the staff Workbench; iteration planning and management of the Kanban board; and the repository governance and CI work that has run through the whole development.

2. Milestone 1: user research, pain points, and user journeys

The research, and our ordering mistake

A sound path would have started from user needs. We started with product design instead: screens, flows, an API. After the first sprint, we began to see how much it mattered to find the real concerns and pain points in the user journey, rather than designing on assumptions behind closed doors. The user research was carried out by the team in parallel, after design had already begun.

The research gave us solid material. We collected 155 valid survey responses, ran industry interviews and user observation, analysed public customer complaints and industry sources, and grouped users into three kinds: people filing their first claim, people who want efficiency, and people in urgent or complicated situations. The central finding: the problem is not just that forms are long. People lose confidence and control during a claim because of a chain of things they do not know: what to do first after the incident, what evidence to collect, whether the insurer has received enough, what happens next, and who is responsible for the next step. With AI in the picture, two more join the list: has it actually understood me, and will a human step in when things get serious. Among respondents who had been through a claim before, about a third said they had not known what information was required; asked what they valued most, nearly half chose clear requirements and simple document upload. What people want is a guided, transparent, continuous process where a human is always within reach. These findings directly support our two core capabilities: ask according to the situation; keep progress, responsibility and the next step visible, and carry the full context across any handover.

The research also exposed our blind spots. It was exploratory: some questions had very few respondents, blank answers can only count as missing data, and the conclusions are strong enough to set priorities but not to stand as population statistics. The evidence on the staff side came mostly from public process material and one limited industry interview, and still needs validation by real claims staff. We kept these reservations in the later design and did not treat exploratory findings as settled fact.

My part: turning the findings into verifiable journeys

The research was mostly my teammates' work; what I did at this stage was translation, turning its conclusions into a verifiable development plan. I broke the product goals into sixteen functional requirements and twelve acceptance scenarios, each one describing a complete, observable experience: a customer describes a rear-end collision and the system produces a claim draft in which every fact carries its source; a customer mentions that someone is hurt and the normal information gathering stops at once and the case goes to a person; a customer comes back a week later and continues from where they left off. The completion standard the team set was: every demonstrated path must genuinely change system state; static demos do not count.

I then had a first go at implementing the early journeys, as a good first issue: guided intake connected to real interfaces, the urgent-interrupt and human-support paths for reports of injury or danger, and the full line from fact capture through staff handover to a status the customer can see written back. Two design decisions from this work have stayed in use ever since: a document that does not exist yet (say, a police report that takes a week to issue) suspends only itself, while everything else in the case that can safely move forward keeps moving; and when a customer explicitly asks for a human, the request is honoured immediately. Both came from real complaints in the research: one missing document holding up an entire case, and being talked out of human help again and again.

3. Milestone 2: from Prototype to Validation Prototype

The Prototype showed that the journeys made sense. But the "Agent" at that point was a fixed script, with no code of behaviour and no data it could digest; the interface and our way of working together were just as provisional. My work in this stage was to push the prototype that carried our design ideas toward the real thing: an Agent driven by a real model, with every step checkable; an interface built on good frontend practice, centred on user pain points and our design position; and a way of collaborating that everyone could rely on.

3.1 The Agent: behaviour first, model last

Before writing the Agent documents and code contracts, I surveyed how mature systems handle conversational service, covering the major dialogue platforms [1][2][3][4][5], insurance-specific products [6][7][8][9][10][11], and the infrastructure that supports long-running processes [12][13][14][15][16][17][18][19][20][21][22]. Vendor material varies a great deal in reliability, so I graded every source: official technical documentation, publicly readable implementations, product claims with no visible internals, and my own inference, and I excluded a portion of the claims that were marketing.

On one point these systems agree: the model handles understanding and expression; the application owns facts, permissions and process. On that basis I designed our own behaviour system. The first step was to pull apart the things a single conversational turn had been mixing together: user intent (what the customer is trying to get done this turn), conversational moves (how to communicate this turn: answer, clarify, confirm or explain), claim commands (controlled changes to the claim record), tool calls (bounded capabilities such as looking up a policy or registering a document), and runtime control (continue, wait, interrupt, hand to a person). The old prototype expressed all of this through eight blended actions; one action was both a reply style and a state change, and there was no way to validate them separately. Pulled apart, a turn now produces a plan with several dimensions: one customer sentence can receive a response, generate several field updates each carrying its source, trigger a policy lookup, and set up the next step, with each kind validated on its own and leaving its own record.

Constraint comes from two places. One is the registries: the fields, actions and tools the system allows are a finite list, each with its own validation rules and permission requirements; the model may only propose within the list, and anything outside it is rejected outright. The other is splitting a claim's "content" from its "progress" into two state machines: content branches answer what the incident involves (a vehicle, a collision, a third party, a police report not yet issued) and can stack; the lifecycle answers how far the case has moved and who it is waiting on, with transitions controlled by the application. The dynamic form is a projection of these two machines: one customer sentence can fill several fields at once, each holding its value, its source and its confirmation state, and a correction keeps the old value in history; the next question is chosen by what the current action actually lacks, and once there is enough, the case moves on rather than working through a form item by item.

The model access layer is its own block, for two reasons. One is cost and availability: we cannot be locked to a single vendor's pricing and outages, so evaluating and swapping models has to be cheap. The other came as explicit advice during this stage: insurance data carries privacy requirements, and a real deployment may not be allowed to send data to a cloud model at all, so there has to be an explainable plan B, a locally deployed model in its place. The design goal of the access layer follows: the behaviour contract does not move; changing models means changing an adapter, and differences in interfaces and field structures are absorbed inside the adapter; each model profile registers the capabilities it has been verified for, and may not take on a use it has not been verified for.

Most of this design has by now landed on the main branch. The behaviour catalogue is a formal document with structural checks, in which each of fifteen behaviours registers its trigger, the results it must produce, the results it must not produce, and its behaviour on failure. The action registry, in five namespaces, is in the backend; every action carries a version, a closed input definition, the permission its execution requires, and a side-effect class. The model access layer supports OpenAI-compatible official, relay and local endpoints, and before every call it checks the selected model profile: identity, protocol, privacy class, prompt version, timeout and evaluation status, refusing the call if anything falls short. The intake paths for all three scenario types are connected to a real model, which uses versioned prompts and strict structured output to extract several source-linked facts from a single customer sentence; a failed turn writes no partial state. The most recent step turned the field catalogue and branch rules for the three product lines into an executable backend contract: motor, home and contents are mutually exclusive product families, a family can only be selected by confirmed facts, and values the model has inferred stay as candidates; after a customer correction the branches are recomputed, and a branch that exits goes into history together with its sources instead of being silently deleted. Every action the Agent touches now also has a uniform audit record format: who initiated it, whom it affects, on what authority, and what actually happened, including failures and unknown outcomes, must all be recorded; the contract is generated from code, and CI watches that it and the documentation do not drift apart. My teammates' work converges on the same line: the access layer's configuration and error handling have been completed, verification of real model providers has been hardened, and policy and claim-history lookups are wired into conversational turns. The boundary that has to be recorded honestly: the repository still runs the controlled script by default, and connecting a real model is an explicit configuration act; the live path has tests for configuration, transport and provider verification and runs on synthetic data, but it is not yet the default path for the three-line demonstration.

3.2 Redoing the interfaces: paying back the research debt

Late in this stage I ran a full E2E inspection of both the claimant side and the staff side. The results were not good, and most of the problems traced back to the ordering mistake of Milestone 1. The claimant landing page had six entry points with no hierarchy, and two of the three claim types led to dead ends. The login journey was broken: after logging in or registering, users were dropped onto an isolated account page whose back button returned to the entry page, not to the conversation they had just left; they could not tell whether login had succeeded, and could not find the claim they had just started. Page elements were piling on top of one another, with marketing blocks, a document-preparation checklist, demo controls and the chat all crowded into the same layer. The guided form put nine fields on one screen, against our own design position of one main question at a time. A fixed "Step 1 of 3" progress bar hung on the page, though the real process does not fit in three steps: cases that go through a human handover or wait on documents have no place in it, so the assurance the bar gave customers was false. Body text sat below the accessibility floor. The staff side was a single 150,000-character HTML file, with key information hidden behind mouse hover where keyboard users cannot reach, and demo controls mixed into production features. None of these problems can be pinned on any one person; they are the debt that accumulates when interface decisions run ahead of user understanding.

I completed the redo, from design to implementation, in a little over two days. The starting point was to turn the research persona into a set of design rules, and to build one shared visual language for both frontends. The image the research produced, sharpened with the help of our industry mentor: a person in a claim trusts a professional who quietly gets things done, and the product should feel like a private client manager. From that I set concrete choices, each with a reason. The base colour is a warm off-white. For the accent I looked at competitors and chose a greyed green: the insurance industry has worn out trust-blue and safety-green, and together they read as nothing but template; distinctiveness comes from restraint, so the accent appears only where it means something, such as a newly confirmed fact. Documents still to come are never marked in red; an unanswered question is not an alarm. Hierarchy is built from background depth, page, sidebar and panel darkening step by step, with no rules or shadows. These choices did not stay in the design file: I collected the colours, font sizes, font weights, spacing, radii, borders and state semantics into a token library shared by both frontends. Component styles may only reference tokens, stray hex values are not allowed, and each semantic has exactly one token. The two sides share one character but keep their own information density: claimant body text is never below 16px, with cognitive load kept low as the priority; the staff side may be denser, but text that carries state or actions may not shrink to the point of being hard to read.

Both sides also moved from single-file pages to a componentised structure, with state, routing and styles each behind their own boundary, so changing one button no longer disturbs another page. Routing carries state protection: refresh, back and a directly typed URL all pass validation first, and the app will not render half a page. One more principle runs through both sides: the interface does not display capabilities the product does not have. Controls like file upload and voice show clearly as unavailable, with the reason, until the backend is ready; what is waiting shows as waiting, what has failed shows as failed, and no static copy dresses anything up as success. The process state is the single source of truth, and the interface is only its projection.

After the redo, the conversation is the single main entry on the claimant side. Overall, the claimant client now feels somewhat like the GPT or Claude web app, but with our own distinct character: beside the chat, a "What we have so far" area lists every fact with its value, source and status, and highlights what this turn added. A customer who wants to change something can edit the field directly, or simply ask the Agent in natural language to correct it. What the model proposes takes effect only after the customer confirms it, under concurrency protection, so a stale page cannot overwrite newer facts. The login journey has been straightened: a claim can be started without logging in, a successful login returns straight to the conversation, a claim begun anonymously is attached to the account, and the account page is reached only through Profile.

The staff side I reorganised around the order of a staff member's work: the first screen answers which claim to handle now, where it is stuck, and what the next step is; the action area shows only the operations the current state allows, with one primary action highlighted at a time; evidence, history and the full conversation open on demand as reference material. Fraud indicators appear only as signals with their sources attached, for professional review, and the word "fraud" never appears on the interface as a conclusion. Handling a claim requires an explicit accept first; accepting is atomic, so two people cannot edit the same claim at once. The staff-side Agent produces only draft suggestions; a draft becomes an action only after the staff member edits and confirms it, and permissions are checked again at execution.

How this redo was settled took as much care as the redo itself. It touched several people's territory, so I wrote a consolidating document that merged six earlier design drafts, stated what takes precedence in a conflict, and tagged every conclusion with a status: confirmed, current fact, or open for discussion. Along the way thirteen decisions emerged that I could not settle alone, including the final visual tone and the layout density, and I turned them into discussion topics for the team. The redo document also states: the frontend does not invent backend fields; a new field starts with a written proposal to the backend owner.

3.3 Governance: teaching a team, and its tools, to collaborate

This line of work has run from the first week to now. It began with an observation: most of my teammates had never worked in a professionally run repository, and practices like branch protection, reviews with substance, and task tickets that state what "done" means were new to them. My approach was to let the repository demonstrate them itself: set up the collaboration rules and templates, add automation so that task cards move with development progress, and nobody needs to hold the whole project in their head to contribute. Later we were honoured to learn that we were the only group of the thirteen to have automated synchronisation between repository tasks and the Kanban board.

Writing the rules down did not make them hold. The turning point was watching a change get approved and merged while every check was failing; the main branch broke, and another change had to repair it. The lesson I took was specific: reminders work on people, unevenly, and hardly at all on AI tools working from partial context, and my teammates all use such tools. So I made the rules executable: one authoritative rules document that both people and tools will read, a local check before push, and a merge gate on the server side that nobody can walk around; later the gate moved forward to the creation stage, so a task ticket that cannot say what problem it solves gets flagged before anyone spends time on it.

Then the infrastructure failed in a way no rule had anticipated. The checks I had made deliberately thorough, plus that board synchronisation, consumed 90% of the monthly quota shared by thirteen groups. The quota burned out in one afternoon, and the administrators disabled the CI service for the whole organisation shortly after. I handled the problem end to end: that evening I tried free virtual machines from two cloud vendors and could not get one running, so I spent the night rebuilding the two things that could not wait, the merge checks and the board synchronisation, on a free serverless platform, debugging along the way a runtime error that only appeared under real events and never reproduced locally; the next day I submitted an organisation access request to the new CI platform and emailed the repository administrator to explain the situation, and once approved, completed the migration. Technically this taught me to rebuild a critical chain under pressure and validate it with real traffic; on the communication side it taught me that anything requiring someone else's approval should be raised at once, rather than letting the repository stall while I tried every technical option myself.

After the incident I thought again about what makes a check worth a team's trust. My answer has four parts, and all of them have since gone into the repository. The evidence must be real: if the remote check did not run, a genuine local run is required, and nobody may claim a passing record that does not exist. Failures must be explainable: a red light nobody can account for only teaches people to ignore red lights. A check must not be bound to a single platform: this incident proved a platform can stop at any moment, so the merge requirement hangs on a platform-neutral status, and switching platforms needs no rule change. And the cost of a check must match the change: I rebuilt the whole chain to run by impact, so a documentation-only commit no longer runs the full backend checks, an ordinary change checks only the affected parts and their direct consumers, and the main branch still runs everything. The checks that once burned the whole organisation's quota now spend effort as needed.

3.4 Planning and the Kanban board

I have managed the planning and the Kanban board throughout, and the method has changed several times as problems surfaced. At first the goals were broken into day-sized tasks with no fixed roles, which left each person scattered across six or eight fragments a day, with tasks blocking one another. Next, each person took stable ownership of one technical stack, product paths ran in parallel across the stacks, interface agreements joined stack to stack, and dependencies that were not ready were stood in for by test data. Later still it became clear that the vertical slices, each advancing on its own, did not add up to a complete result a user could see, and acceptance was unclear, so the method changed to task-as-goal: one task corresponds to one feature that can be accepted whole, and each capability carries a status sheet marking it verified, partially available, not open, or dependent on test data. "Done" went from a single word to a graded statement, and a demo can no longer quietly stand on a test double everyone has forgotten.

4. Professional growth and reflection

Understand the business before designing for it. I treated insurance as a backdrop: in the first week I helped design a product whose claims process none of us really understood, and the research we caught up on later, the current reporting process, the field-level requirements, the existing fraud controls, was homework that should have come first. The interface inspection later read like an invoice: behind every problem was a place where the interface had made an assumption on the user's behalf. The habit I take away: before designing any surface, be able to say which research-confirmed fact each element answers to.

The real depth of GitHub collaboration. Before this project I thought I knew Git and GitHub; what I actually knew was how to use them alone. The problems that only appear when five people work in parallel, I hit for the first time in this project. Task ownership: a task assigned to A that B quietly finishes produces resentment and authorship nobody can untangle, so we later required that remaining work be transferred only with a written statement of clear boundaries. Review chains: with five people reviewing at once, any single merge can void every other review in progress, because the versions they pointed at are gone; I learned to sequence merges the way one sequences dependent tasks, and to require that reviews target the current version. CI is a shared and finite resource, and the quota incident taught me that designing a check chain is also designing a budget. Another lesson came from the platform's own limits: it can only identify accounts, not whether a person or a tool is operating behind one, so rules can only constrain behaviour and must give up guessing identity. Last, the discussion area: it is the right place for proposals that need a decision before any task should exist.

What a good Agent is actually made of. Having finished this stage, I can say where the engineering effort in this work really sits: the model is the smallest component in the system. Around it, we need a behaviour boundary that does not depend on prompts; we need "the model proposed, the system approved, this actually happened" recorded as three separate things; we need a closed catalogue of fields and actions, with nothing outside it available for the model to invent; we need a vendor-neutral access layer, so the model can change while the product's behaviour does not; we need error handling that distinguishes retry-safe, non-retryable and outcome-unknown, with duplicate-execution protection wherever a side effect exists; we need safety checks that run before the model speaks, because signals like injury are not the model's call; we need retrieved documents and customer uploads treated as data, never as instructions; we need a source and a confirmation state on every fact; and we need evaluation of the whole handling process, because looking only at the final sentence misses the overreach and the errors in between. Not one of these comes with the model. Compressed to one line: agent engineering is using the model's fluency without inheriting its authority.

Working with people. The governance work changed how I communicate. Early on I corrected teammates' process mistakes directly, which made me the bottleneck and, at times, the villain. What worked better was letting the system give the feedback: a check that fails with a written reason teaches the same lesson without a face attached. Person-to-person discussion is kept for genuine design disagreements, and the topics list from the interface redo became my template: state what needs deciding, list the options honestly, and hand it to the team.

5. Next steps

The coming stage turns the MVP into a Validation Prototype. We want to prove two things: that the intake can adapt to the situation and earn the customer's trust, and that customer, Agent and staff work on the same claim record. The demonstration will cover three product lines, and the test doubles in the runtime paths will retire in favour of real services. I am responsible for the Agent's behaviour and prompts, the dynamic form, the claimant interface, and the Agent side of the process state machine. Journals 3 and 4 will cover that stage and draw toward the final report.

References

[1] Google Cloud, "Dialogflow CX: Pages," https://cloud.google.com/dialogflow/cx/docs/concept/page [2] Microsoft, "Generative orchestration in Copilot Studio," https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-generative-actions [3] Genesys, "About Genesys Agent Copilot," https://help.mypurecloud.com/articles/about-genesys-agent-copilot/ [4] Intercom, "Fin Procedures explained," https://www.intercom.com/help/en/articles/12495167-fin-procedures-explained [5] AWS, "Amazon Connect Cases," https://docs.aws.amazon.com/connect/latest/adminguide/cases.html [6] Hi Marley, "Claims," https://www.himarley.com/claims/ [7] Sprout.ai, https://sprout.ai/ [8] Five Sigma, https://fivesigmalabs.com/ [9] Shift Technology, "Claims solutions," https://www.shift-technology.com/solutions/claims [10] Indemn, https://www.indemn.ai/ [11] AWS Samples, "Serverless EDA insurance claims processing," https://github.com/aws-samples/serverless-eda-insurance-claims-processing [12] Rasa, "Flows," https://rasa.com/docs/reference/primitives/flows/ [13] LangChain, "LangGraph durable execution," https://docs.langchain.com/oss/python/langgraph/durable-execution [14] Temporal, "Workflow execution," https://docs.temporal.io/workflow-execution [15] Open Policy Agent documentation, https://www.openpolicyagent.org/docs/latest/ [16] AWS, "Amazon Bedrock Knowledge Bases: retrieval configuration," https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-config.html [17] AWS, "Amazon Bedrock Guardrails," https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html [18] Microsoft, "Security trimming in Azure AI Search," https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search [19] LiteLLM documentation, https://docs.litellm.ai/docs/ [20] OpenAI, "Structured Outputs," https://developers.openai.com/api/docs/guides/structured-outputs [21] Anthropic, "Building effective agents," https://www.anthropic.com/engineering/building-effective-agents [22] Anthropic, "Effective context engineering for AI agents," https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents