Field notesResearch
Nobody records who caught the failure
Every record in our failure ledger has a field called detected_by. It is meant to answer the most operationally useful question you can ask about an incident: what found this?
Across 4,308 records, 97 of them answer it.
Where the other 4,211 went
3,719 records carry unknown. The field was never filled.
491 carry api. That one looks like an answer and is not. Our write endpoint sets detected_by: source || 'api', so api is what gets stored when the caller supplies no source at all. It records the pipe the record arrived through, not the thing that noticed the failure. A default that looks like data is worse than a blank, because a blank invites a question and a default closes one.
One record is null. That leaves 97 with a value somebody chose: a monitor, a session, a person asking why the emails had stopped.
Why this is the expensive field to lose
Knowing what broke tells you what to fix. Knowing what caught it tells you whether your detection is worth what you pay for it.
Those are different questions and only the second one compounds. If every failure in a quarter was caught by a person looking at a screen, you do not have monitoring, you have a staffing arrangement that happens to be working. You will not find that out from a list of root causes. Every one of them will look like it was handled.
We built checks, gates, canaries and health probes across these systems. We cannot say from this ledger how many failures they caught, because the field that would tell us is empty on 86 percent of records and defaulted on another 11.
We can say something about it from a sample. We drew 401 records stratified by month and coded them by hand against a written codebook, including what the prose says actually surfaced the problem. That gives detection rates with real confidence intervals, and it is in the Agent Failure Index. But a sample of 401 is what you do when the census field failed, not a substitute for it having worked.
The shape of the mistake
This is not a story about one missing column. It is the normal way an observability field dies.
The field was optional. Nothing rejected a record without it. The endpoint supplied a plausible default so no write ever failed for want of it. And no report was built that would have looked wrong if the field were empty, so nobody noticed it was.
Any one of those three on its own is survivable. Together they guarantee the outcome. The field existed for five months, accumulated 4,308 opportunities, and produced 97 usable values.
What we changed
The default is the first thing to go. A write with no source should store a null, not the string api, because a null is honestly missing and api is quietly wrong. We would rather have 3,720 blanks than 3,719 blanks and 491 records that look answered.
The second is that the field stops being free text. What detected something is a closed list: a check, a person, the client, a later unrelated failure, or an audit that went looking. Five options, one of which must be picked.
The third is the only one that actually enforces the other two. We now report detection mix monthly. A number that someone reads is the only thing that has ever kept a field alive, and we have four months of evidence that nothing else does.
What to ask
When someone shows you their incident history, ask what fraction of it names the detector. If the answer is most of it, ask to see the distribution, because a field that is 90 percent one value is usually a default.
If the answer is that they do not track it, that is at least honest, and it is where we were until we counted.
The dataset, the method and what it cannot tell you are published in full at the Agent Failure Index.