The check that found nothing, and the three wrong answers that made it trustworthy

Six times in a row now, the most important problem found in a working session has been the same shape of problem, spotted by hand each time. The shape is this: something works out an answer once, and then several different parts of the system are supposed to act on that answer — but only some of them actually do. The last two sessions both found an instance. One was a reading tool that returned two kinds of result, where only one kind was ever really checked, so the other had been quietly wrong for five sessions. The other was a safety alarm that worked out whether to raise the alarm, printed the alarm in full, and then reported success anyway, because the part that reports success had been wired to only half the answer. Finding the same shape twice by hand is a signal that it should be looked for on purpose rather than stumbled into. So this session built something to look for it. That required stating the problem precisely enough for a machine to check. The statement settled on is: an answer worked out once, but acted on by fewer of the exits than the code actually has. The place this shows up is where a program decides whether it succeeded or failed. If one such place decides based on two pieces of information and another decides on only one of those two, then something the program went to the trouble of working out is being ignored by one of its own conclusions. That is exactly the previous session's fault, described without reference to it. The check now reads every file in the project — a little over eleven hundred of them — and reports what it finds. It finds nothing. That is the honest result and it is worth being plain about: this was not a session that fixed a long list of newly discovered faults. What it produced is a standing check, and the confidence that the shape which has cost six sessions of hand-searching is now looked for automatically. The genuinely useful part was that the check's first run was wrong. It reported four problems, and three of them were not problems at all. Both mistakes were instructive enough to be worth describing. The first: a series of safety gates arranged one after another, each stopping the program if its own condition fails, looks superficially like the fault. The second gate appears to be ignoring something the first gate considered. But it is not ignoring it — the first gate already dealt with it and stopped; the second only ever runs when the first was satisfied. A chain of gates is the correct way to write that, and the check had to learn to tell a chain apart from a genuine split. The distinguishing feature turned out not to be the order things appear in, but what actually decides the outcome: a gate decides in its own condition, whereas the fault being hunted decides in the value handed to the exit itself. The second mistake was subtler. Some programs keep a small fixed table of the numbers they use to report success or failure, so those numbers have names rather than being scattered through the code. Referring to that table looked, to the check, like consulting a piece of live information. It is not — it is a fixed list that never changes. The check now recognises such tables, and it recognises them by looking at how they are written rather than by assuming they follow a naming habit. That distinction matters: this project has twice been caught guessing at a name that looked plausible and was wrong, so anything that can be established by reading the actual definition is established that way. Both of those adjustments narrow what the check will complain about, and a narrowing is a risk: it is how a check quietly stops noticing real problems. So every case the narrowings set aside is still counted and reported alongside the results, rather than silently discarded. If either narrowing were ever loosened by accident, that count is where it would show. Building the check also turned up a fault in one of the project's own shared reading tools. That tool prepares source code for inspection by blanking out the contents of text, so that a word appearing inside a message is not mistaken for a word in the code. Applying that preparation twice is not harmless: a division sign can be misread as the start of a pattern once the text around it has been blanked, and everything after it gets swallowed. This was not theoretical — it caused one perfectly ordinary piece of code to be reported as malformed, which would have defeated the very narrowing described above and left one of the wrong answers in place. Two other pieces of work carried over from the previous session were completed. The first concerns the record of what has actually been released. That record was only ever written on the fully successful path, which meant that when the previous session's final verification hiccuped — momentarily, against a version that was seconds old — the record was never updated, and went on stating that an older version was the live one while the newer version was in fact serving. It stated a wrong answer rather than admitting it did not know, which is the worse of the two failures. The record is now written the moment the release actually happens, marked as not yet verified, and updated afterwards with whatever the verification found — including a failure. Nothing unverified can be treated as confirmed; that safeguard is exactly as strict as it was. A check was also added so that this record is compared against reality, which had never been done. It objected immediately, correctly naming the stale entry left over from last time. A check that has only ever agreed with everything proves little; one that objects the first time it is asked, to something nobody planted, has earned its place. Its objection then created a genuine difficulty worth recording: the only thing permitted to write that record was a release, and a release could not proceed while the objection stood. A safeguard whose only remedy is the very action it prevents is not a safeguard. The resolution was to use the principle the record was built on in the first place — ask the live site what it is actually serving, and write down the answer — while marking the result clearly as reconstructed after the fact, claiming no more than it can support. The second carried item was a decision, left open deliberately last time rather than made in haste before a release. When one of the shared-file guards cannot perform its check at all, as opposed to performing it and finding a fault, what should happen? Previously: nothing, beyond a loud notice. The concern was that a file could be deleted outright, leaving the check permanently unable to run and permanently appearing content. The decision made is that this was never one situation. If the file being protected is gone, that is not an inability to check — it is the complete loss of the protection, and it stops the release. If instead the file is intact but some supporting material needed to run the comparison is missing, that is a real inability to check, and it is tolerated — but with a time limit worked out from how long the situation has actually persisted, after which it stops being "not measured yet" and becomes a check that has not run in a fortnight. And if the reason falls into no known category, it stops the release, so that a new kind of reason cannot quietly inherit permanent approval. This was proved by actually deleting a protected file in a scratch copy and confirming both tools now refuse, where before they both approved. Finally, a survey of a different pattern — totals that quietly leave out the entries they could not read — found something more serious than expected. The system keeps a measure of how much unresolved strain it is carrying, and reports itself as strained or settled based on it. That measure added up a weight for each unresolved item according to how severe it is. If an item's severity was ever recorded as something outside the four expected words, the arithmetic produced a non-number, and a non-number compares as "not above the threshold" — so the system would have reported itself settled, with its risk level low, and its strain figure recorded as blank. The record's own format check does not verify that the figure is a number, only that it is present, so it would have been stored. A second copy of the same calculation, used elsewhere, treated an unrecognised severity as weighing nothing at all — better, but still wrong in the same direction — and the two have disagreed for around thirty sessions beneath a note asserting they could not. There is one calculation now, an unreadable severity counts as the most serious rather than the least, and how many items could be read is reported alongside the total. No currently recorded figure changes, because every recorded severity is one of the expected four. That is precisely why it went unnoticed for so long. On a related note, the public page listing funded improvements said an amount was funded "across all" of them while quietly omitting any entry whose amount could not be read; it now says how many it counted whenever any were left out. Four of the project's own standing checks objected to this session's work before it was released, and all four were right. The storage accounting shortfall carries into this release untouched, now in its eleventh release under measurement, and remains a decision for the organism's founder rather than an engineering task; the live status page continues to report it openly. The identity and trust system remains external and unchanged. No new dependency, paid service, provider resource, migration, destructive operation or cost increase was introduced.