实习日志 / English / Control plane: which model, rules, knowledge and data did this turn use?

Control plane: which model, rules, knowledge and data did this turn use?

中文 | English

By early September, more and more of the project had become "configurable": which model to call, which protocol adapter to use, which knowledge base version, which version of the Agent's rules and prompts, whether data ran locally or on MongoDB, and which third-party services were connected. Each of these could be changed on its own, yet together they decided how the Agent behaved in a single turn. I started to worry about one situation: the runtime reads new rules, an old model and a different knowledge index all at once, something goes wrong, and nobody can say what that turn actually used.

On 6 September I wrote a full build plan for a control plane. The question it had to answer is the one in the title.

What it manages, and what it does not

The control plane manages system configuration, knowledge, models, Agent rules, external integrations, evaluation and runtime state. It has to guarantee six things:

• everything that can be released has a version;
• changes go into a draft first and are published only after validation;
• the runtime reads only published versions;
• publishing, rollback and failed operations are fully audited;
• administrators cannot change the state of a live claim through the control plane;
• configurations never end up in a mismatched combination of versions.

The last two draw a boundary. Claims, work items, human handoffs, customer messages and claim facts belong to claims operations. The control plane manages "how the system runs" and never touches "what happened on a particular claim". Its API offers no path to write claim state, work items, handoffs or claimant messages directly.

A set of objects

The plan defines ten core objects:

The key one is the Release Set. Individual configurations can be edited separately, but a change that spans several domains cannot be activated piece by piece. The final release pins a set of mutually compatible versions in one Release Set, for example:

Model revision: 7
Data profile revision: 3
Knowledge revision: 18
Agent rules revision: 12
Integration revision: 5
Feature revision: 9

The runtime loads only complete Release Sets and records the version of every domain in it. That gives two layers: the version of a single domain (say, Agent rules revision 12), and one complete release that combines them. New rules with an old model and yet another knowledge index can no longer happen, by construction.

The configuration domains number eleven: system, model, data profile, knowledge, Agent rules, external integrations, access, evaluation, feature flags, runtime state and audit. The data profile domain lists fixture, local, Cloudflare, MongoDB and AWS runtimes, and the control plane also checks their compatibility.

Lifecycle and approval

Every version goes through:

draft -> validation -> published -> superseded

plus a few side paths: a failed validation goes back to draft; a draft can be withdrawn; a rollback is a new publish operation; every accepted or rejected operation produces an audit record.

The rules are direct. A published version cannot be edited; to change it you create a new draft. A failed validation cannot change what is currently published. A version without complete validation evidence cannot be published. A rollback creates a new release record and never overwrites history. Failed operations are audited too.

I kept approval deliberately light. A normal-impact change can be published once it validates. A high-impact change needs an independent review record from someone other than the author before it is published, but there is no approval queue and no approval workflow; the publish endpoint only checks that the record exists. A five-person prototype team needs the fact "a second person looked at every high-impact change" to be recorded. It does not need a full process.

Third-party services: capability and task are separate

Third-party services exist in the system in two very different ways, and the plan puts them on two sides.

On the control plane side is the Integration Registry, which records what a service can do and how to connect to it: service identity, type and environment; supported capabilities and their versions; adapter type, endpoint reference and region; which data fields may be sent and at what privacy level; which consent is needed and which Northwind authorisation; the secret reference; timeout, retry, idempotency and the policy when the outcome is unknown; connection checks, health and the latest failure reason; whether it may enter a Release Set; and its own version, publish status and withdrawal history.

On the claims operations side are external tasks and requests, which record what was requested on this particular claim: the request, the consent, the sending status, the provider's reference number, result verification and safe retries. The control plane does not rewrite these claim-level records.

Account management without a second authentication system

The control plane also gives an entry point for managing customer and staff accounts: search, create, deactivate, restore, edit the fields that may be edited, manage staff roles and scopes, reset or revoke sessions, and view the audit of account changes. It does not copy the authentication system. It keeps using the repository's existing token library, opaque bearer tokens, server-side token hashes, and role and scope checks. Account operations go through the existing backend repositories and database boundaries. Credentials such as passwords and tokens are handled only by the existing identity service; the control plane API never returns them in plain text and never accepts identity fields submitted by the browser as grounds for authorisation.

Every write goes through authentication, scope, If-Match and Idempotency-Key, returns stable error codes, allows no partial commits, and appends to the audit.

How the runtime connects

Connecting the runtime to the control plane means only a few things: fetch the current published Release Set; load all of it; check that model, knowledge, Agent rules and data profile are compatible; and record the version source in every turn, every model call and every retrieval. There are also three prohibitions: drafts, unvalidated or superseded versions must never be used as an implicit fallback; when a provider fails, the runtime may only switch to a standby configuration that meets the same boundaries; the runtime must never publish, modify or downgrade configuration by itself.

With this in place, every claim, every Agent turn and every audit record can answer the same question:

Which model, rules, knowledge and data configuration did this behaviour use?

The admin console shows only what is real

The admin front end calls only the management API. It cannot connect directly to the database, object storage, model endpoints, vector index or secret manager. It shares the tokens and state semantics of the claimant and staff apps and may be denser. Its layout follows real admin tasks and backend projections, not an old prototype filled with data. Components show only the states and actions the API returns and never derive publish, permission, connection or third-party state in the browser. Publish and rollback buttons appear only when the backend returns the corresponding allowed action.

One rule in the plan I wrote especially firmly: a provider that is not connected yet must show unconfigured, unsupported or its real failure state, never a fake "connected", "published" or "healthy". Asynchronous operations such as import, indexing, evaluation and connection checks must have real Operation Records, never a static progress bar. Release Set, Runtime Snapshot and Integration Registry can be marked as delivered only when the backend model, storage boundary, API and validation evidence are all in place.

So the plan opens by declaring what it is: a target specification, not a description of the current state, and no promise that every module in it is already built. Implementation and acceptance are checked item by item against "design requirement, API, persistence, front-end display, real validation", and whatever is missing goes into a gap list.

The final chain

Administrator creates a draft
  -> system validates configuration and dependencies
  -> validation evidence is produced
  -> a high-impact change records one review by someone other than the author
  -> a complete Release Set is created
  -> atomic publish
  -> runtime loads it and records the versions
  -> results are traceable
  -> a problem leads to a rollback publish
  -> every step goes into the audit

The three key designs added by this plan are the Release Set, cross-domain release consistency and the Runtime Snapshot. Without them the control plane is a set of admin pages. With them the system can actually be published, rolled back and traced.

Later

"Never use an old version as an implicit fallback" came up again in a redundancy audit in early October. The audit found that when no Release Set had been published, the system quietly assembled an old-style model configuration from environment variables; when the runtime was missing a configuration, it silently fell back to an old prompt; and the admin form for creating a release configuration had its "runtime environment" field pre-set to fixture, an in-memory test environment that loses everything on restart. All of this went against the principles in the plan. It grew bit by bit, in many "let's just get it running" moments.

After that cleanup, a runtime missing required configuration refuses to start and says which item is missing, and the admin drop-down defaults to the configuration the server is actually running. Writing a principle into a plan does not guarantee it is kept. It takes repeated checks to find where things have drifted.