> ## Content Index
> Fetch the complete content index at: https://khalidalsuwailem.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Close to Perfect, Useless in Practice
- URL: https://khalidalsuwailem.com/writing/close-to-perfect-useless-in-practice/
- Published: 2026-08-02T17:37:27.000Z
- Updated: 2026-09-02T17:37:27.000Z
- Description: A component that scored close to perfect on its own tests and was useless against real outcomes. On the gap between what you can measure and what you can claim.
- Author: Khalid Alsuwailem
- Tags: Evaluation

The component scored 0.98 on its own test suite. It was, by the numbers, done. Against real outcomes — the thing it was supposed to predict in the world — it was no better than a rule that always said no.

## How a test suite lies

It doesn't, exactly. It answers the question it was asked. The problem is that the question drifted: the suite was built from the failures we understood, and the system was deployed against the failures we didn't.

### Three ways this happens

- **Leakage** — the label is in the input, somewhere you didn't look.
- **Selection** — the test cases are the easy ones, because easy ones are the ones you can label.
- **Proxy drift** — the metric was a stand-in for the outcome, and the two came apart.

## The through-line

This is the same problem as marketing attribution. A platform's conversion number is a proxy that scored perfectly on the platform's own test. The *gap between what you can measure and what you can claim* is not a bug in one field; it is the shape of measurement itself.

---

I find that gap more interesting than either field on its own.