实习日志 / English / CI Governance / CI, part two: 77,000 credits in a month, and a table of "which line is covered by which test"

CI, part two: 77,000 credits in a month, and a table of "which line is covered by which test"

中文 | English

The previous post covered the night our Actions quota ran out, ending with the move to CircleCI. The move did not end things: two weeks later we found CircleCI credits being eaten at the same rate. This post covers the month after, how we worked out where the money went, and how we finally got CI to be cheap and still genuinely protect quality.

The first time: 30,000 credits in half a month

In early September the team used about 30,000 CircleCI credits in roughly half a month. The repository had no billing breakdown by job or by PR, so I could not split that number into exact contributions per task, but several things clearly stacked up.

The CI topology was bigger than the product. The pipeline carried backend, claimant, staff, admin, automation, docs and PR rule jobs at once, and three of the front-end jobs still targeted old front-end surfaces, some of them early single-file HTML prototypes. They had long stopped being product entry points, yet every run installed their dependencies, built and tested them. A few more jobs look "stricter", but every job costs real money; an abandoned surface left on the pipeline keeps charging on every trigger, and makes people think it is still a supported product.

The same thing was verified several times. One PR's changes could satisfy the triggers of both GitHub Actions and CircleCI; a small backend change started the full backend test suite, format check, type check and contract check; a change to the CI configuration itself was treated as a backend behaviour change and triggered the whole backend suite. The most hidden case: a job that finally decided to "skip" had already been created, pulled its image and started, and the credits were already spent.

So the first conclusion I wrote down then was that if skip inside a job does not save money; real pre-filtering must happen before the job is created. Consumption can be understood as a product:

total consumption ≈ start-up and run cost per job
                  × jobs created per pipeline
                  × pipeline triggers
                  × dependency install and test duration

That also set the order of optimisation: first cut jobs that should not exist, then pipelines and jobs that should not be created, and only last tune the speed of individual tests.

Two steps: deduplicate, then pre-filter

Step one was deduplication. The three old front-end quality jobs moved into the archive and out of the formal workflow. Every job that stayed had to be able to say which product or governance behaviour it verifies. The PR rule check is the safety boundary of collaboration, so it keeps running everywhere and is never filtered by path. The full quality set on main lost nothing.

Step two was pre-filtering. CircleCI used a setup pipeline with path filtering: first read which paths changed relative to main, set parameters such as backend_changed, docs_changed and ci_changed, and then create only the matching workflows. Inside the backend quality job, a selection script picked the relevant test files from the changes: ordinary backend behaviour changes ran only the affected tests; changes to shared domain models, repository protocols, runtime assembly or the dependency list ran everything; changes to docs, CI configuration and the selector itself did not escalate to the whole backend suite, only to a light check that parses the YAML and runs the selector's self-tests.

I also wrote a list of "things you may not do to save money": remove the full quality suite on main; turn business tests into a check that the YAML parses; replace real API and contract verification with fixtures; let a job start and then pretend with a shell condition that it "did not run"; send arbitrary unisolated PR code to a long-lived host with sensitive configuration just because a self-hosted runner is ready; or report a local pass as a CI pass without remote evidence.

An execution strategy

Right after that I wrote the strategy as eight points, centred on "the fastest feedback, the lowest cost, without coarsening the tests":

1. Narrow the PR gate: non-backend changes do not trigger backend tests; an ordinary backend PR runs only the unit tests of affected modules, the contract and auth tests of affected endpoints, core tests that cannot wait such as permissions and claimant visibility, and lint and type checks on affected files. Running everything on PRs by default was the main cause of credit consumption at the time, and is forbidden.
2. Run the full suite less often, but do not cancel it: only on merge to main, nightly, and when shared core files change.
3. Shard only for main and nightly full runs. Sharding shortens wall-clock time but increases credits: 4 shards go from about 56 to about 200 credits, so on PRs it neither saves money nor solves the real problem.
4. Use a static import dependency graph for impact analysis instead of a hand-written file mapping.
5. Deduplicating tests is the real cure, with one deletion rule: delete only cases that cannot show they add behavioural coverage; never cut just because there are many.
6. Putting a flaky test in quarantine requires several conditions at once, and tests for permission boundaries, claimant visibility, idempotency, revision conflicts, append-only audit and external service authorisation may never be downgraded or skipped for being flaky.
7. Cache the whole environment keyed on the lock file hash. Expected savings are tens of seconds to a minute or two; it is not expected to fix the main problem.
8. When a branch gets a new commit, cancel its older queued and running pipelines automatically. Zero cost, zero risk, done first.

Since everyone on the team used AI coding tools, tests were being generated far faster than people could review them, so I also wrote test-writing conventions for Agents: before writing a new test, ask whether this is a new business behaviour or the same logic with a different input value; a single parameterised test function has no more than 8 to 10 cases, and beyond that each parameter must state which different business contract it verifies; unit tests mock external dependencies by default, and tests that hit real dependencies must be marked @pytest.mark.integration; and do not tell tests apart with _1, _2 suffixes.

The second time: 77,000 credits in a month

On 3 October I ran a full CI audit. The data came from CircleCI's statistics for the last 30 days, per-step job timings and pytest output, the status of the last 40 merged PRs, and timing every one of the 269 test files on main individually on my own machine.

The result surprised me a little: in the last 30 days the quality workflow ran 2,091 times and used about 77,000 credits, 2.5 times the previous crisis.

Half the problems the audit found had already been thought through in September and had not held.

Pre-filtering had slipped back

The setup pipeline approach was tried on 9 September, but the configuration on main had reverted to "create all 8 jobs on every push, and decide inside each job whether to skip". github-automation-quality had no path filter at all and ran npm ci, checks and a wrangler bundle every time, while only 2 of the 1,687 commits on main in the last 30 days touched its directory; that job alone used 9,474 credits. The project had not turned on "Only build pull requests" either: 22% of pipelines ran on branches whose PR was not yet open or already closed, and another 4% on branches that never had a PR. Agents tend to push after every step, and that is where this cost came from. About 40% of the credits went on jobs that should never have existed.

The main reason for slowness: 2 wasted seconds per test

backend-quality spent nearly all its time in pytest: a PR waited a median of 4.9 minutes, p90 20.4 minutes, at most 34.1 minutes. PRs that selected 600 to 1,400 tests took 9.5 to 33 minutes, which works out to about 0.85 seconds per test on CI.

Timing file by file made the reason clear. Of 3,860 tests, 2,710 were very fast, under 0.1 seconds, together only about 1% of test time; the other roughly 1,150 took 1 to 3 seconds each, 98% of the time. Almost all of that group used the client or app fixture or called create_app(...) themselves. Breaking down one app lifecycle:

The second second was pure waste. I wrote a pytest plugin that changed only that wait to 0.02 seconds, without touching a line of business code: one TestClient lifecycle went from 1.01 seconds to 0.03, test_account_data_api.py from 90 seconds to 36, and test_claim_api.py from 156 seconds to 73. Estimated across the 1,150 slow tests, this one point could cut total backend test time roughly in half.

The selector was pushing Agents to write new tests

This was the most interesting finding of the audit. The selector's rule: if a changed file matched the hand-written mapping, select the corresponding tests; a backend file that matched nothing selected only the selector's own self-tests, 68 of them, entirely unrelated to the change. And CI required that the selected tests cover 85% of the changed lines.

I simulated each of the 251 backend files on main: 140 were not in the mapping at all, so changing them ran only those 68 selector self-tests; 37 pulled in 469 to 671 tests on any change; backend/domain/models.py, the shared core model, should by policy trigger everything, and actually pulled in only 148.

In terms of how Agents work, it happened like this: an Agent changes a service file that is not in the mapping, CI runs only the selector self-tests, and changed-line coverage immediately reports 0%. The easiest way out is to create a new test file that walks through the changed lines; since the new file is "in the change", it gets selected and coverage passes. The repository may already have had three test files covering the same logic, but they were not selected, so they did not count.

That is why test code grew in step with backend code:

By early October, tests/ held 111,109 lines of Python and the backend 112,612, roughly 1 to 1.

I annotated the audit report at the time: isn't 85% changed-line coverage standard practice? Surely the problem is how the AI responds to it, not the threshold itself? On checking, the threshold was fine; what was wrong was that it only counted "the tests selected this time". So what really had to change was the selector, plus the rule for AI: reuse first.

There was also a counter-intuitive finding: copy-paste duplication was actually rare. After AST-normalising test function bodies, only 19 groups, 45 functions, were structurally identical and at least 8 lines long. Low-quality tests were mainly of other kinds: tests keeping dead code alive; test files named after "which PR, which week, which issue", such as test_p17_*, test_week5_*, test_issue_928_*; tests in the tests/fixtures/ directory that pytest collected but the selector never chose; and one topic spread over many files, such as 6 files for unfinished claim recovery and 8 for retention. All of them came from the same mechanism: every PR opening a new file to satisfy coverage.

There was no longer a full run on main

This last one was my own doing. A commit on 17 September removed CI's full mode, and main was changed to test only the HEAD^1..HEAD diff. Removing full mode was my decision, made to save credits. The result was that no run on main ever ran all the tests again. Running file by file this time found 8 test files failing on main, and one of them was precisely the test asserting "CI must contain a full pytest command". It had been failing since 17 September and was simply never selected.

"Test only the diff" has a precondition: something, somewhere, has to run everything. Both documents I wrote in September said "main still runs the full quality suite, and it may not be weakened to save credits", and the configuration had already drifted away from both.

The audit had two more points where my view differed from the report's recommendation. The report said main had no required status checks, so CI was only advice; my view was that CI was never meant to be a gate. The report said 5 GitHub Actions workflows pointed at self-hosted runners that did not exist, with 138 of the last 200 runs failing to start, and called that pure noise; those were the fallback for when credits run out, so it is normal that they were not triggered, and I chose to ignore that item.

Landing it: a table of "which line is covered by which test"

A governance PR went up the same day as the audit, plus one follow-up patch. After merge, things changed like this:

Create only the jobs that are needed. .circleci/config.yml became a setup pipeline, where scripts/ci_plan.py first works out which workflows to create and then continues into continue_config.yml. github-automation-quality runs only when its own directory or .github/scripts/ changes. "Only build pull requests" was turned on in project settings, with main as the exception.

That 1-second wait. The in-memory repository's event listener switched to 50-millisecond polling.

Selecting tests by coverage. This was the core change. After merges to main and once a day, a full run in 4 shards with --cov-context=test builds, from the coverage data, a table of "which test files cover which line", and caches it; PRs read this table to select tests. The hand-written mapping and the "sentinel" self-tests were deleted together. Tried twice on main's real table: changing one line of one function in config.py selected 7 test files, where the old logic would have run the 68 selector self-tests; changing one line in messages.py selected 11 files, where the old logic needed 16 files and 667 tests. Changed-line coverage can now be satisfied by tests the repository already has, so PRs no longer need new tests just to pass the threshold, and the incentive to pile up tests is cut off at the root.

Full-run results. On the PR, 3,846 tests passed and 0 failed, each shard taking 8.5 to 11.4 minutes, with 89% total coverage; the pipelines on main passed too. The daily full run is set at 14

The tests that had been failing. Of those 8 files, all but one had outdated assertions: the design had changed, a version number had moved, a schema had gained a field, middleware had been swapped. The remaining one looked like a real bug; it got its own issue and was marked xfail(strict=True). Along the way I also found a test that globally replaced sqlite3.connect, which, with coverage contexts on, dragged down every test after it; that was fixed too.

One place for PR rules. PR rules used to run twice, once in the Cloudflare Worker and once in CircleCI. The commit format check moved into the Worker and CircleCI's pr-policy job was deleted.

The rule became "reuse first". The repository's Agent rules, governance skill and PR template all changed: new tests go into the existing test file of the module under test first, new files are named by module or behaviour, and CI only advises rather than blocks; the parameterisation limit went from a hard rule to a recommendation.

Landing it hit three snags. The 90% total coverage floor in pyproject.toml applied to every shard, so it had to be turned off for shards and for PR test selection. The scheduled trigger passes a nightly parameter into the continuation config, and an undeclared parameter is an immediate error, so another PR was needed. And the Worker could receive a PR's opened and edited events out of order, so a newly created PR briefly showed the rule check failing, recovering after one edit to the description.

Still to do: let tests share one app (each test still spends about 1 second in create_app), merge test files by topic, and delete tests that keep dead code alive; the last one waits to be handled together with a separate dead-code audit.

Between the two crises

Putting the two side by side, the first time's conclusions were all right: pre-filter before jobs are created, keep the full run on main, never weaken evidence to save credits. The problem was that they lived only in documents. A configuration change, one decision made to save money, and they quietly slipped back, with nothing to raise an alarm.

What the second time taught me in addition was about incentives. 85% changed-line coverage is a reasonable threshold, and an Agent writing new tests to pass it is a reasonable response; stack those two reasonable things on an inaccurate selector and you build a test suite as large as the business code. So this time the fix was to make CI itself accurate, to let it know exactly which existing tests cover each line of code. Rules are only explanations written for people and Agents; when CI measures accurately, it stops pushing people in the wrong direction.