Frontier · capability appraisal
Engagement 001 · commissioned by nobody
116 commits · four days · 18,552 lines
Git remotes configured: 0
An assessment of how this software is built, against how software is actually built. It does several things better than most shipped products and one thing worse than any of them: it has never been given to anybody.
§1
This is not a courtesy paragraph. Four of these are practices that most professional teams talk about and do not do.
| Practice | Measure | Industry norm |
|---|---|---|
| Comments that record why | 5,862 of 16,850 lines — 34.8% | 10–20%, and mostly restating the code |
| A regression check per fix | 4 check tools, 1,556 measured pairs | usual, but rarely written from the failure |
| Checks that fail without the fix | one pins the old behaviour | rare — most tests only assert the new |
| Determinism by construction | seeded RNG, versioned save, drop-on-mismatch | common in engines, rare in gameplay code |
| Measure before optimising | 123s → 0.24s, both numbers recorded | usually asserted, seldom timed |
| One source of truth | no sidecar JSON, on purpose, argued in the file | the argument is normally lost |
| Append-only record | 70MB, sealed in segments, nothing deleted | telemetry, but usually sampled and expiring |
The comment culture in particular is not decoration. Nearly every fix in this repository is accompanied by a paragraph naming the wrong behaviour, the measurement that revealed it, and the reasoning that produced the new one. That is a discipline most codebases claim and almost none keep, and it is the reason an outside assessor can audit four days of work in an afternoon.
§2
§3
These are not industry failings borrowed. They were designed here, from scratch, and each of them has a mechanism.
Eighteen of 116 commits touch ad/. The deliverable of a debugging session
is a typeset document with a colour palette. Seventeen notices exist for a program that
has zero configured git remotes: the marketing department is four days ahead of
the distribution department, which has not been founded.
The ritual is now stable. No Go for Launch disclosed the project was not ready. Fear and Loathing opened by noting that the machine writing it had said the night before that it was fixed. Ninety-Two Heads concluded ship it, do not ship it as a game. This appraisal exists because the question was asked a fourth time. Nobody has ever changed their answer, and the answer has never changed anything.
Six faults were found in one sitting. The three found from inside the record — reused identifiers, a lost update, a goal counter — were fixed within the hour. The three found by a person looking at the screen — workshops producing nothing, a population frozen at fifteen, a store full of one commodity while another rotted outside — had been running in plain sight for sixteen thousand ticks. The author reads files. Files do not contain the thing.
Twenty assertions in the record checker. Every one is about file segments, markdown parsing and identifier arithmetic. Not one asks whether a settlement has ever produced a good. The suite would have passed, green, throughout the entire period in which this valley manufactured nothing.
Twice in the same file. nobodyElse means anyone may build when no
builder is free — and becomes everyone is a builder, always in a
settlement with no builders. theirs guards hand-hewing for foresters, and
in a settlement with two foresters and 263 logs it forbade the other eight from picking
up a log they were standing on. Both conditions are correct for the common case and
exactly backwards for the case that actually occurred.
The codebase's own comments name this sin three separate times — a menu that listed eleven of twenty-nine actions, a five-building list found "for the third time", a tool ladder that existed twice. A list that has to agree with another list will stop agreeing, and this project keeps writing the second list anyway.
The premise of the whole project is that the record is trustworthy and nothing in it is
ever deleted. It has no run identifier. Open a second browser tab and a brand-new valley
writes its opening lines — "t":0, two settlers coming ashore —
into the middle of a run at tick 14,462. Both are still in there, interleaved, with
nothing to separate them. This was found today and has not been fixed.
§4
The engagement asked for a number. The number is eleven, and here is the arithmetic, so that it can be shown to have been wrong.
| Notices to date | 17 | in 44 hours — one every 2.6 hours |
| Commits per notice | 6.8 | 116 commits, 17 notices |
| Faults open at publication-blocking severity | 5 | output, population, run identity, CI, second reader |
| New faults revealed per fault fixed | 0.67 | this sitting: five closed, over three new opened |
| Fixes to drain the queue | ≈15 | 5 ÷ (1 − 0.67), which assumes the ratio holds |
| Notices per fix | 0.6 | this sitting: three notices, five fixes |
| Estimate | 11 | and the repository is pushed somewhere on notice 28 |
Two caveats, both material. The first is that the discovery rate has never once fallen below the fix rate: every sitting closes faults and opens roughly two-thirds as many again, which is what a convergent series looks like right up until it is not one. The second is that this estimate does not include the notices that will be written about this estimate having been wrong, and on the evidence of the previous four attempts at the same forecast, that is not a small correction.
There is also a shorter answer. Publishing requires exactly one command, and it has never been run, and no notice was ever needed to explain why not.
§5
The previous notice reported that the client's messages were becoming less hinged and more accurate, and treated that as a joke with a chart in it. On reflection it is the most important finding in this appraisal, and the joke was concealing the mechanism.
Here is the mechanism. Every fault this office found, it found by reading a file that the program had written about itself. Every fault the client found, they found by looking at the program. Those are not two levels of effort at the same task — they are two different tasks, and only one of them can see a kiln that nobody walks to.
The capitalisation is not noise around the signal. It is what a person sounds like at the moment they see something the instrumentation cannot say, and this office should have learned by now to treat a rising voice as a defect report with a severity field attached. Every one of the client's exclamations on this record — the depot, the workshops, the joiner, the fifteenth settler — named a real and previously unreported fault. The office's own prose over the same interval grew steadily more composed, better typeset, and found nothing that could not be found sitting down.
Assessed finding. The project's most effective diagnostic instrument is a human being with no access to the source, watching it run, getting annoyed. It has no budget line, no test id, and it is not in any of the four check tools. It is also the only thing here operating at level five.
§6
| Do this | Cost | Catches |
|---|---|---|
| One check that asserts a settlement produces a good in N ticks | an afternoon | the entire class of faults this project cannot see |
| A run identifier on every logged line | one field | two valleys interleaved in one record, silently |
Replace npm test with npm run check | one line | the next person, who will type npm test |
git remote add, then push | one command | seventeen advertisements finding an audience |
| Watch it run for ten minutes without reading anything | ten minutes | on the evidence, more than any of the above |
Overall capability rating
Level one. The craft is at level four and the process has no floor under it.
A team that writes 5,862 lines of reasoning to accompany 10,988 lines of code, and pins
every fix with a check that fails without it, and then leaves npm test as the
placeholder npm wrote and never gives the program to a second person, has not built a bad
piece of software. It has built a good one in a room with the door shut.