Engineering notes. Mostly about what a design was actually for, and what happened to it afterwards.
A verification gate returned a detailed, specific, well-evidenced failure. It named the tests. It was wrong, and nothing in its output could have told me so — a verdict cannot detect the condition that invalidates a verdict. The guard for it already existed, wired into one of the three places that needed it.
A broken automated check announces itself. A check whose baseline is corrupted keeps passing, keeps proving it ran, and quietly asks for less — and every summary statistic you have will say the run went well. Two of these in one night, and the five wrong answers I chased first.
A two-term model of per-step wall clock predicted the tool was at human speed. Running the benchmark suite end to end falsified it at 3.6×, and the third term reconciles two measurements that had looked contradictory for a week.
I froze a decision rule before the tool it judges had produced a single verdict, including one clause I was fairly sure would never bind. The first result came back perfect on the headline metric — and failed on the clause, because the gate cannot see the thing the model was quietly deleting.
A 7B model on a laptop wrote unit tests at 0.85 of Claude Haiku’s survival rate for zero API tokens — and the 14B, at twice the time and twice the memory, did not do better. The decision rule was written down first, and the result landed in a hole in it.
I built a system around the idea that a worked example is what makes a small model usable, then ran the ablation that was supposed to confirm it. 4 of 20 became 16 of 20 on a single sentence about an import statement.
ps is not a memory
measurementmacOS reported 205 MB resident for a model whose weights are 4 GB, while the machine held 9.5 GB of swap. Under pressure every number you would reach for moves the wrong way — including the one you would use to decide whether there is a problem.
Six times a measurement said a model was bad at its job, and six times the model was fine. A field guide to the failure that looks exactly like the result you were expecting.
A prototype from May 2022, the platform release that superseded it a month later, and the three things that still transfer.
Perception for an agent should be an always-on daemon, not a function the model calls. Measured on the iOS Simulator: a screenshot loop at 210 ms per look against a framebuffer daemon at 6.74 ms, and a ten-step flow from ten model turns and ten images to one turn and none.
On the simframe site
Why four national authorities reach opposite conclusions from the same premise.
On the cbomctl site
NIST IR 8547 scopes its 2030 deprecation by security strength, not by algorithm.
On the cbomctl site
What actually made AI coding agents useful on a 220,000-line codebase.
On Medium