实习日志 / English / Agent Control / Context is a resource: where 9,000 tokens per turn came from

Context is a resource: where 9,000 tokens per turn came from

中文 | English

By mid-September the claimant Agent could hold a conversation, but it was painfully slow; a very simple question sometimes took more than eighty seconds. At first I thought the prompt was too long and planned to split it into dozens of small files. Before doing that I measured, and found the problem was much more serious than "a long prompt".

Measure first

On 16 September I measured one ordinary conversation turn on main, using the real request assembly logic and a minimal test claim:

Tokens were estimated as characters divided by 4. Each provider's tokeniser will differ, but the order of magnitude makes the point. And this was an almost empty test claim with no real conversation history.

Broken down, the fixed cost came from:

• the v6 main prompt at the time, about 2,440 tokens;
• the shared output schema, about 1,532 tokens;
• a duplicated "runtime contract" the code appended after the prompt, about 614 tokens;
• every ordinary turn being forced to call the model twice. The first call carried the full prompt, the claim context and a claim.read tool, asking the model to read the claim first; the second sent all of that again, plus what was read and the full output schema. Yet most of what claim.read returned duplicated the claim information already injected in the first call.

Some things were not in the table but were going to blow up sooner or later. History was cut at "the last 12 messages", and a single message could be up to 5,000 characters, so history alone could approach 60,000 characters. Clause snippets returned by the knowledge base had no size limit. The first call carried tools and the second did not, so with the tool set changing, the provider's prefix cache would rarely hit. Finally, the Bedrock adapter at the time rejected ordinary tool calls while this path required claim.read, so the whole flow could not run on Bedrock at all; calling it "provider-independent" was simply not true.

After measuring, the idea of splitting the prompt into dozens of files fell apart. With the same total content, sending it all in a different folder structure saves not a single token. What had to change was what actually went into the model each turn, and how many calls were made.

How others manage context

The next day I went through how 12 kinds of public systems handle context, mostly from official documentation and engineering posts published by the teams.

Kiro's steering files have four loading modes: always loads every time, fileMatch loads when the current file matches, auto loads by relevance between the description and the request, and manual loads when the user explicitly references it. They express four kinds of control over when loading is triggered, unrelated to prompt length. But this is only a trigger classification; using it directly as the permission model of a business system would be wrong.

Cursor also has four kinds of rules. It recommends keeping rules focused and small, referencing authoritative files instead of copying them in full, and not putting into prompts what a linter can check mechanically. Its skills expose only a name and description at startup and load the body on demand; subagents use separate context windows for noisy work such as code search and long logs, and the main Agent receives only a summary. The docs also warn that subagents have a startup cost, so simple tasks get slower.

Claude Code treats CLAUDE.md and memory as context, not enforced configuration; to block an operation mechanically, the docs recommend a hook. Files imported with @path are expanded in full at startup, so splitting into several imported files only organises things more clearly and does not lower the context cost at all.

Codex reads from the user's global rules down to the AGENTS.md in the current directory when a run starts, merging up to 32 KiB, and does not rescan during the run. It suits stable repository constraints, not business state that changes every turn.

Aider does not send the whole codebase to the model. It first sends a very small repository map, files, key classes, functions and signatures, with a default budget of about 1,000 tokens. From the map the model understands the main interfaces and asks for specific files when it needs detail.

Sourcegraph Cody's docs have a line I agree with: too little context misses key code, too much slows responses. What to aim for is the density of relevant information; "the smaller the better" was never the goal.

Anthropic's context engineering recommends preloading stable core information, fetching long-tail information just in time, compacting overlong history and moving stale tool results out of the active context. It also warns that compacting too aggressively deletes information that only later turns out to matter.

Manus's engineering post treats KV cache hit rate as the single most important metric for a production Agent: keep the prompt prefix stable, since even one token of difference can invalidate everything cached after it; make serialisation deterministic and ordering stable; append rather than rewrite where possible; do not add and remove tool definitions frequently inside the Agent loop; and restate the current goal at the end of the context to counter "the middle gets ignored". This is one team's experience, not a cross-model protocol, but it agrees with the prefix caching constraints described by OpenAI and AWS.

OpenAI Agents SDK automatic compaction keeps blocking for several seconds after the last output token; when low latency matters, the docs suggest turning it off and compacting manually between turns or when idle. Google ADK has two compaction triggers, by token threshold and by a fixed number of turns, and when both are configured the token threshold wins.

MCP's tool specification requires servers to return tools in a deterministic order, and tool visibility should reflect the credentials and permission scope of the current request. A model seeing a tool does not mean anyone can run it: discovery, authorisation and execution are three different boundaries.

Amazon Bedrock processes its cache in the order tools → system → messages; if an earlier layer changes, every later cache is invalidated, so when the tool list changes, the system prompt and message caches go with it. Cache hits are also not guaranteed and must be checked in the returned usage; for some models, writing to the cache costs more than normal input, and it only saves money if enough is read back later.

Put together, a few shared patterns emerge:

Knowing it exists and sending it to the model are two different things

The most important sentence from the research went into the middle of the diagnostic report:

"The Agent knows some content exists" and "this turn sends its body to the model" must be two separate actions.

Taking that sentence further, a piece of content goes through five states from being known to being executed:

discoverable -> selected -> loaded -> authorised -> executed

• discoverable: the system knows that a prompt fragment, tool, service or reference exists;
• selected: this turn's planning thinks it may be relevant;
• loaded: its body or schema actually enters the model request;
• authorised: the current user, current claim, consent state, visibility and release policy allow it to be used;
• executed: the runtime has checked the specific parameters and actually performed the read or the side effect.

An earlier state never implies a later one. Semantic matching can discover the damage assessment service, but that does not disclose claim data, obtain consent or send a request; a model choosing a tool cannot bypass the runtime's execution authorisation. Collapsing these five actions into one is the common root of context bloat, Agents overstepping, and behaviour nobody can explain.

Kiro's four modes cover only "how loading is triggered"; a business runtime also needs bounded reference resolution, history compaction and isolated execution. So we used seven loading modes:

The boundary of auto_candidate matters most: it can only produce candidates. A vague similarity score cannot pull a large body of text in, let alone grant permission, consent or side effects. Low confidence or conflicting candidates lead to clarification, never to a wider loading scope.

v7: a context runtime that loads on demand

I wrote the definition of v7 in one English sentence, which stayed at the top of the design document from then on:

A versioned, route-aware and budgeted Agent Runtime that selects a stable request profile and loads only the prompt fragments and contextual facts required for the current turn, while keeping discovery, loading, authorisation and execution separate.

The pipeline of one turn

Authenticated message
  -> Deterministic interrupts
  -> Turn Router
  -> Candidate Catalogue
  -> Context Planner
  -> Request Profile Selector
  -> Prompt Composer and Cache Planner
  -> Model Gateway
  -> Proposal Validator

Each stage does only its own job. Deterministic interrupts handle danger, injury, identity, revision conflicts and explicit consent actions first, and templated results do not call the model. The router only decides product family, task and route, and generates no answer. The candidate catalogue only lists what can be discovered this turn, with no bodies. The context planner decides what is inlined, what appears as a reference and what is omitted, and records the reasons and budget. The request profile selector picks one of a few published profiles. The prompt composer compiles fragments in a stable order and separates the static prefix from dynamic content. The model gateway only executes requests that have passed the budget check and never quietly appends anything. Finally the proposal validator has the last word on actions, fields, consent, permission, revision and side effects.

No stage may bypass the previous stage's selection by "including everything, just to be safe".

Prompt fragments

Prompts are split by responsibility into four folders, core, family, task and capability, and each fragment has its own contract in manifest.json:

{
  "fragment_id": "family.motor",
  "version": "v1",
  "path": "family/motor.md",
  "kind": "family",
  "load_mode": "route_match",
  "applies_when": { "product_family": ["motor"] },
  "requires": ["core.authority", "core.provenance"],
  "conflicts_with": ["family.home", "family.contents"],
  "priority": 100,
  "max_tokens": 420
}

The composer picks one family fragment, one task fragment and zero or more capability fragments; resolves dependencies recursively and rejects cycles and missing fragments; checks conflicts (the motor, home and contents fragments never enter together); compiles in the fixed order core → family → task → capability; checks the budget per fragment and in total; and finally produces a stable prompt_bundle_id, for example claimant-v7:motor:intake:external-support. From then on the logs record this bundle ID and the exact list of fragments instead of a vague prompt_id.

There can be dozens of files, but an ordinary turn should ideally load only 4 to 6 short fragments. The value of splitting finely lies in choosing accurately, reusing and seeing clearly, not in making the model read more.

Routing without a model

Routing is a deterministic algorithm. Safety interrupts run first. If the claim has already confirmed its product family, that family is used. If not, the branch registry and a controlled vocabulary identify candidate families; with exactly one candidate it is chosen, and with several or none the route becomes "family undecided", where the runtime asks the claimant, in one fixed sentence, which kind to report first, without spending an extra model call to guess. Once a formal claim has a family, words about another family in this turn's message do not re-route it; cross-family content means opening or switching to another claim, so the current claim's prompt and field contract are not polluted. The routing result carries a model_required flag; when it is false, the runtime returns a deterministic reply directly and makes no model call at all.

One model call per ordinary turn

The design of "force a read of the claim first, then answer" was dropped. The claim has already been loaded and authorised at the authenticated message boundary; having the model read the same object again through a tool only repeats the transfer and adds a round trip. The ordinary path became:

runtime determines the route
-> assembles the minimal prompt and context
-> one model call
-> runtime validates and applies the proposal

Only paths that genuinely lack data, such as checking evidence, searching history or looking up clauses, may make a second call. The output schema was also split into several narrow ones: ordinary answer, field update, third-party service suggestion, evidence operation, human handoff, claim creation. Fields the system owns or can derive are no longer required from the model.

The budget was set as hard numbers:

3,000 is the hard ceiling for ordinary interaction, and the targets are well below it. The experience goal is P50 under 5 seconds and P95 under 10 seconds. Model configuration still keeps a loose technical timeout of 90 to 120 seconds, but only so occasional slow requests are not killed by mistake; it may not be used to hide performance problems. An ordinary request over 10 seconds is recorded as a performance failure.

How each kind of context is loaded

• Claim: inline only the claim ID, revision, family and current workflow state; the fields the current action needs; this turn's latest message; facts that just changed or conflict; what is still missing, what comes next and whether the claim can be created; and the allowed field values.
• Conversation history: stored in four layers: authoritative claim state, a validated rolling summary, the last 2 to 4 turns in full, and range references for older messages. Ordinary turns send only the summary and the last few turns; compaction is triggered by token threshold, with turn count only as a helper. The summary cannot override claim state or promote unconfirmed words into facts, and every summary must be checkable against the original. Compaction runs in the idle time after the reply, so the claimant never waits for a summary model.
• Evidence: by default only ID, type, processing status and one sentence the claimant can see. The body of images and documents is loaded only when the latest message explicitly mentions it, the current task is looking at evidence, the model supports that media, the budget allows it, and visibility and retention rules allow it.
• Third-party services: first filtered by family, intent, service status and consent state; an ordinary turn inlines at most three candidates, each with only the service identifier, its purpose for the claimant, supported actions, the consent scope needed, current status and next step.
• Knowledge, clauses, past claims: reading a whole document by ID is not allowed; a query or selector is required, with a default of no more than 300 tokens per result and at most 3 per turn.

Long content enters the model as a controlled reference:

{
  "ref": "evidence:ev_123",
  "type": "damage_photo",
  "summary": "Rear bumper damage photo",
  "version": 2,
  "available_operations": ["inspect"]
}

To see the body, it is resolved through context.resolve(ref, selector, max_tokens), and every call must give a selector and a limit; the model cannot ask to "read everything". References are generated by the runtime for the current user, claim and turn; the model cannot invent IDs or use a reference to cross a claim, customer, visibility or consent boundary. There is a balance here, though: an extra model call should not be added unconditionally just to save a few tokens. When the runtime can already tell that some content is necessary for the task, it resolves it before calling the model; only when the model discovers mid-task that something is missing does it go through a tool call.

Over budget, stop

Each turn's budget is allocated by priority:

P0  latest message, permission boundaries, output schema
P1  family and task fragments, claim facts the current action needs
P2  conflicting facts, current consent or external task status
P3  recent conversation, matching services, attachment metadata
P4  history summary, knowledge snippets and other resolvable references

P0 to P2 cannot be silently truncated. If they exceed the budget, the runtime returns CONTEXT_BUDGET_EXCEEDED with a safe deterministic clarification. It would rather stop than hand the model an incomplete set of authoritative facts. P3 and P4 can be turned into references, paged or reduced in number, but this is recorded in omitted_sections.

Stable request profiles

Tool definitions fail at both extremes. Sending the full tool registry with every request wastes context and widens the range of mistaken calls; adding and removing tools for every fine-grained route changes the tool-definition segment of the prefix, lowers the cache hit rate and makes provider behaviour hard to reproduce. So only a few stable request profiles are published:

One profile version binds exactly one schema, and its tool definitions, order, schema and static prefix must be byte-for-byte stable. Routing only decides which profile to use and what to inline or reference; it never generates a new tool list on the spot. Whether a specific call may run is still authorised by the runtime by user, claim, permission, consent, visibility and revision. Every model configuration must also declare explicitly what it supports: how structured output works, whether it can call tools, how tool results continue, which media it supports, whether caching is implicit or explicit, and the minimum cache length. A configuration that does not meet a profile's needs is rejected at selection time, instead of degrading after the request has gone out.

Making the cache actually hit

The logical layout of a request is fixed as:

provider-required tool/schema segment
global core authority and safety
cache checkpoint A
family/task/capability fragments
cache checkpoint B, optional
dynamic Claim projection
retrieved context
recent conversation
latest claimant message
Turn Focus recitation

Static first, dynamic after. Providers require different physical orders of tools, system prompt and messages; this is the logical layout, and the model gateway's adapters turn it into each provider's cache checkpoints.

The final Turn Focus borrows Manus's approach: restate at the end only the few short fields that are easily weakened in a long context and that affect this turn's output, the claim revision, the route, the current goal, the next required item, the range of fields that may be changed, and any pending consent or external action. It does not repeat whole safety rules or the history summary.

Nor should useless prompt text be added just to reach OpenAI's or Bedrock's minimum cache length. Whether "shorter but uncached" or "slightly longer but reliably cached" is faster and cheaper has to be compared with measured data.

A visible budget

Every call records data without sensitive bodies: which release, request profile, prompt bundle and fragments, schema and route were used; the loading decision and reason for each candidate resource; raw input, uncached input, cacheable, cache read and write, cumulative turn total and output tokens; how many model and tool calls; first-token and total latency; and whether the 10-second target was met. Raw tokens stay under the 3,000 hard ceiling, and a cache hit is no licence for unlimited growth; uncached tokens and cache reads and writes show the real cost and latency; the cumulative turn total stops a turn from looking compliant per request while its continuations add up to something out of control.

Qwen, GPT and, later, Bedrock are reported separately. A cross-provider average must not hide one adapter's high latency or failed caching.

Checked against the industry

After the design was written, I checked it point by point against public production practice from Anthropic, LangChain and Manus. The broad direction held: a fragment registry with on-demand loading corresponds to LangChain's "Select" and the progressive disclosure of Anthropic's skills; bounded references correspond to Anthropic's just-in-time retrieval; rolling summaries plus recent originals correspond to compaction; deterministic routing follows "if deterministic logic can solve it, do not bring in a model"; and never truncating P0 to P2 is, at heart, protection against context poisoning.

But the comparison showed several gaps, one of them in direct conflict with best practice:

The first draft exposed tools dynamically by route. The intent was to avoid the model choosing wrongly among too many tools, but "adding and removing tool definitions per route" is exactly the pitfall Manus describes: the cache breaks, and the model gets confused when a tool present last turn disappears this turn. The table of stable request profiles above was the result of this comparison: tool definitions stay stable, and "can this actually be called this turn" is checked in the runtime's execution layer.

The budget table did not separate cached and uncached tokens. Cached and uncached prices can differ by roughly ten times. If the prefix is unstable, even a small total can cost more and run slower than a longer design with a stable prefix. So the budget and logs gained the cache fields.

Observability covered cost but not quality. LangChain sorts context failures into four kinds: poisoning (wrong information reused as a reliable fact, for example a summary promoting "the claimant mentioned" to "the claimant confirmed"), distraction (irrelevant content diluting attention), confusion (several similar candidates exposed at once, such as motor and home fragments loaded together) and clash (sources contradicting each other, such as the rolling summary disagreeing with claim state). The logs therefore gained a few quality counters: summary inconsistent with state, conflicting family fragments, stale references resolved, and false-positive candidates.

Key state should sit at the end of the context. That became Turn Focus.

Deep exploration should be isolated, not given more calls. If a task like "compare several pieces of evidence" ever really comes up, the right approach is an isolated subtask that explores tens of thousands of tokens in its own context and hands back a conclusion of one or two thousand tokens. Raising the per-turn call limit from 2 to 4 or 5 is the wrong direction; it would bring back the very bloat being removed.

Which step first

The implementation order followed "see clearly, then cut":

1. Add an observation mode to the old version first, recording prompt, schema, context, history, tools, actual tokens, cache and latency per provider, to form a baseline.
2. Delete the duplicated contract the code appended, and drop the forced claim.read on ordinary turns. This step is the most certain and gives the biggest latency gain.
3. Set up registries for request profiles and provider capabilities, so the tool and schema prefix stabilises first.
4. Add the fragment manifest, candidate catalogue, deterministic routing and layered cache layout, first running with behaviour equivalent to the old version.
5. Add context planning and budget reports, first comparing "planned input" with "current input" in shadow mode without affecting users.
6. Enable fragments, narrow schemas, hard budgets and complete loading-decision logs.
7. Introduce rolling summaries, bounded history references and background compaction.
8. Enable isolated execution for cross-claim, deep evidence and long material tasks.
9. Release the full v7, keeping the old version as a versioned rollback target that takes no part in any automatic fallback.

Acceptance was also counted separately: the ordinary path calls the model exactly once, 100% of the time; input P95 stays under 3,000 tokens; total latency P50 under 5 seconds and P95 under 10 seconds; no duplicate claim reads; no unused tools, service catalogues or family fragments; stop when over budget, never truncate silently. The static prefix of one request profile at one version is byte-identical and its fingerprint reproducible; tools that are discovered, selected and loaded but not authorised are never executed.

One more point I wrote into acceptance on purpose: if input size and call count already meet the targets and Qwen's P95 still cannot get under 10 seconds, that is clearly classified as a problem of model throughput or deployment capacity. Performance gains must not be faked by deleting more necessary business information.

Back to the nine thousand tokens

This changed how I look at prompts. A prompt is only one kind of resource in the context; the claim, history, evidence, field contracts, third-party services and tool definitions should all follow the same rules of on-demand loading and budgets. The goal of the optimisation was never the fewest tokens per turn. It was to make the ordinary path call the model once, keep information dense and the cache prefix stable, all while staying correct, authorised and traceable, and to make every extra load explain its reason, source, budget and authorisation.