SPECIMEN · ML · · 1 min read
The Guard I Couldn’t Build
A guard I couldn’t build without breaking legitimate inputs, and a detection rule that turned out to be statistically indistinguishable from chance.
The task was to put a small model in front of a larger automated system and have it refuse the inputs that would make the big system fail. Simple to state. I spent two months on it and shipped nothing, which is the interesting part.
The shape of the problem
Every guard is a classifier, and every classifier has a threshold. Move it one way and you stop the bad inputs along with a slice of the good ones; move it the other way and the guard becomes decorative. The question is never "does it work" but what does it cost the people it was not built to stop.
Attempt one: the rule
A hand-written detection rule, tuned on the failures we had seen. On the held-out set it was fine. On a fresh week of traffic it flagged legitimate inputs at a rate I could not justify, and when I sliced the flags by source they were indistinguishable from a coin toss.
Attempt two: the model
Better numbers, same disease. It learned the surface of the failures, not the mechanism, and the mechanism is what matters when the inputs drift.
The failure of a monitor is invisible by definition. If it were visible, you would not need the monitor.
What I took from it
I publish this because it didn't work. The discipline around a guard — how you test something whose whole job is to notice when things go wrong — turned out to be the real project, and I am still on it.