实习日志 / English / Journals / Journal 4: Results, Reflection and Application of Knowledge

Journal 4: Results, Reflection and Application of Knowledge

中文 | English

1. Project Results

Overview

Northwind FNOL Agent handles the First Notice of Loss (FNOL), the first report a customer makes to an insurer after an incident. The insurer in the brief is Northwind Insurance. The claimant describes what happened in their own words. The system turns that account into a claim record in which every fact has a source and can be checked, and carries the claim forward to the next safe step. Claims staff pick up the same record and do not have to ask the claimant again. The project consists of two connected key functions: a claim intake that adapts to the claimant's situation, and a single claim record shared by the claimant, staff and external services.

Functions and Specifications

The system has five parts: the customer client, the agent and runtime, the staff workbench, knowledge and external services, and administration and operations. I was responsible for the agent's behaviour and prompts, context loading and the division of responsibility inside the runtime, and the customer client experience. Later I also took on the triage conditions, the list of questions awaiting an answer, and the case panel. As the repository maintainer I set the goals and tasks for each iteration and maintained the team's collaboration norms. On 8 October the project was frozen and presented to industry judges at the poster presentation.

Customer client. A claimant can start a claim without signing in and describe the incident by text or voice. A claim continues across sessions, so someone who comes back a few days later carries on where they left off. The claim form grows as the claimant talks. It starts with the few fields every claim needs. When a rear-end collision is mentioned, vehicle fields appear, and when an injury is mentioned, injury fields appear. Three product lines are covered: motor, home and contents. Each fact records which of the claimant's sentences it came from and whether it has been confirmed, and the claimant can check and correct it directly. The system accepts photos and PDFs and reads the document type and key information from them, such as the amount on a quote. When a document is not yet available, only the steps that depend on it are put on hold and the claim keeps moving. When the claimant asks about policy wording, the answer cites the clause it comes from. Anything the system cannot answer, or should not answer, goes to the claims team, and the claimant sees who they are waiting on. Before submitting, the claimant reviews a summary. After submitting, they receive a claim number and the next step.

Agent and runtime. The model understands the claimant and phrases the replies. The runtime owns facts, permissions and process. Every model output is only a proposal, and it is executed only after the runtime has validated it. The proposal, the approval and the actual result are recorded separately. Fields, branches and claim stages are versioned, closed lists, and the model can only propose items that are on them. The context sent to the model is loaded in layers by product line and task. When there is a sign of danger to a person, the runtime interrupts the normal flow before the model is called and hands over to a person. Model access is a separate layer. It can use a cloud service or a local model deployed on the team's server, and this can be chosen per session.

Staff workbench. A submitted claim enters the staff workbench, where fixed conditions sort it into Fast-track, Standard or Complex and the reasons are shown. The result is recalculated when the claimant supplies new documents. A staff member taking over a claim first sees the items the claimant still owes and the questions awaiting an answer, and continues with the full context without asking again. Where documents do not agree with each other, for example a receipt dated after the theft, the system raises a reminder for staff to judge. It does not conclude that there is fraud.

Knowledge and external services. Policy wording is held in a versioned knowledge base, and retrieval filters by product line, version and effective period. External services such as repair booking, damage assessment and towing receive requests and return results within the same claim, and information is shared only after the claimant consents.

Administration and operations. Prompts, model profiles, the lists and the knowledge base are published with versions through the admin client. The runtime only references published versions, so the version used in any turn of a conversation can be traced afterwards. The backend is written in Python (FastAPI). Data is stored in MongoDB and files in object storage. The customer client, staff workbench and admin client are three separate React front ends, and the first two share one design system. The deployment target is AWS, and model access already has an adapter for Amazon Bedrock.

2. Reflection: What I Learned from the Project

Understand the business before designing. At the start of the project we went straight into designing the product. The interface, the flows and the system interfaces all came before we understood the claims business. How insurers take claims today, what a claim needs to collect, and which fraud checks already exist were researched only later, and that research should have come before design. A full review of the customer client and the staff side followed. Behind every problem it found was an assumption the interface had made on the user's behalf. The landing page had six entry points with no priority among them, and a page carried a "Step 1 of 3" progress bar that the real flow did not fit. Feedback from industry judges at a stage presentation pointed the same way: the claims process has to stay close to how claims really work, and an interface that only looks like claims does not show that the product solves the problem. The practice I took away is that before designing any interface, I should be able to say which researched fact each element responds to.

Design is always centred on the user. A good experience has to be tested from the user's position. On one hand, only by walking through the system from start to finish in a real browser could we see, as a user would, where our design went wrong. On the other hand, our user journey was once thrown out and rebuilt, because it took the feedback from sprints and presentations for us to realise that our design had drifted away from the real situation. The mindset I formed is to put myself in the user's situation at that moment before judging any feature, and when building a feature, to think as the user first and ask what I would really want.

Communication in teamwork. Each of the five of us built a different part, and communication is what lets those parts connect. Both sides of an interface understand it the same way, problems are found while they are still cheap to fix, and less work is redone. I learned two main practices. The first is to say the whole thing and write it down. The opening states the current situation, what I need from the other person and by when, so they can respond as soon as they finish reading, and a written message can be checked again later. The second is to be specific. Feedback points to where things do not match and attaches the evidence, and a problem is raised early, because the earlier it is raised, the more room everyone has to adjust. The same matter is also told differently to different people. With teammates I go down to interfaces and fields. With judges and supervisors I talk about value and outcomes.

Working with people. The hard part of working together is that everyone holds different information and comes from a different background. Some teammates know the backend well, some work on data, some build the staff interface, and the same sentence can mean different things to each of them. I learned to find out the other person's goal and constraints before discussing a solution, and to give enough background that they can pick the matter up directly. On division of work, we agree on boundaries and interfaces first. Within each person's area I respect their judgement on how to do it, and anything that crosses a boundary is discussed before anyone acts. I treat an opinion as a claim to be verified, whether it comes from a teammate or from my own review. When I find I was wrong, I say so and correct it on the spot, because a wrong judgement left in place will be followed by whoever picks the work up next.

Research at every stage. Research has to be repeated as the project moves through its stages, and every step should rest on enough of it. It draws on mature industry practice and on real data and documents, and its purpose is to find the users' real pain points and the real difficulties in the design before a direction is set. Sources also need grading. A vendor's claim about effectiveness is treated as a product description, a finding from a small sample is used only to set priorities, and a public review counts as a single example. Outside opinions and my own first ideas go through the same check against evidence before they are adopted.

3. Applying What I Learned to the Internship

Software engineering. The basic software development process runs through requirements analysis, design, implementation, testing, deployment and maintenance. The output of one stage is the input of the next, so an upstream mistake is amplified downstream, and the later it is found, the more it costs to fix. In the course project for Software Tools and Techniques (COMPSCI 732), my teammates and I built a complete piece of software by following this process. In the internship it guided me in two ways. When a problem appears, I first work out which stage it belongs to and go back to that stage to fix it, and I do not patch it downstream. Each iteration also runs from requirements through to a result that can be accepted, so testing and integration are never left to the end.

Artificial intelligence. Advanced Topics in Machine Learning (COMPSCI 760) covers how models are trained and evaluated, and a model's ability can only be stated through its performance on evaluation data. This led me to evaluate the model in the project over complete claim conversations, because looking only at the final reply misses the overreach and errors in the middle. Health Knowledge Management (DIGIHLTH 702) covers how knowledge is organised, updated and governed. It led me to treat policy wording as knowledge that needs governing. It carries a version and an effective period, and when the knowledge base has no clause for a question, the system says so and passes the question to the claims team.

Human-centred design and research. The experiment design, interviewing and qualitative analysis taught in Advanced Topics in Human Computer Interaction (COMPSCI 705) guided our initial exploratory research, which included a survey and user interviews. The course also taught how to obtain informed consent. Before a study begins, participants receive a participant information sheet and a consent form that explain the purpose of the study and how the data will be used, and that taking part is voluntary and they can withdraw at any time. The team's survey followed this. It was anonymous and voluntary, and a participation statement came before the questions.

Security, privacy and ethics. Usable Security and Privacy Engineering (COMPSCI 725) taught me to ask two questions first when designing: how could this system be misused, and which of the customer's information is sent where. The project handles personal information such as incidents, injuries, photos and policies, and the questions shaped two parts of the design. The model can be switched to a local model deployed on the team's own server, so customer information does not have to leave the country. The fraud reminder relies only on inconsistencies between documents and does not use personal attributes such as age, ethnicity or address.

Systems thinking. System Dynamics and Complex Modelling (OPSMGT 741) covers feedback loops, delays and system archetypes. Two of its ideas, "fixes that fail" and "a local optimum is not a global optimum", gave me words for what happened to the agent's conversation. Each round of patching resolved one specific conflict when looked at alone. Stacked together, the patches kept producing new conflicts inside the loop, and the conversation grew stiffer with every change. The loop itself had to change. The approach was to return to the goal, define what a good claim looks like first, and then work backwards to the design decisions it needs.