Our failure recurrence rate is 2 percent and we do not believe it

Across 4,308 recorded production failures, 82 are marked as having happened more than once. The highest recurrence count on any single record is 6.

That is a repeat rate of 1.9 percent. If it were true it would be a remarkable number, and we do not think you should believe it. We are not sure we do.

1.9%

What we checked

A low recurrence rate has two readings. Either the same failures genuinely are not coming back, or the thing that detects a repeat is not detecting them.

In version 1.0 of this study it was the second one. The ledger fingerprints each failure and increments a counter when a match arrives. We swept 359 records for duplicates and found pairs with byte-identical titles that had both been filed as first occurrences. The fingerprint was missing.

We ran the same sweep over 4,308 records. Normalised exact-prefix matching found zero duplicate pairs. Token-overlap matching at a 0.75 threshold, which catches rewordings the first pass misses, also found zero.

So the fingerprint that was broken is now working. Repeats are folding into the counter instead of opening new records, which is why the sweep is clean and why 82 records carry a count above one.

Why that still does not make 1.9 percent a real number

A fingerprint matches a failure to one that looks like it. It cannot match a failure to one that shares a cause and presents differently.

That is most of the repeats that matter. A configuration default that is wrong in one place is usually wrong in four, and each will surface as a different symptom in a different system on a different day. Three separate records, three separate root causes, one underlying mistake made once. Nothing in our tooling joins them, and nothing in this dataset would show you that it had happened.

So 1.9 percent is the rate at which we re-file a failure that looks like one we have already filed. It is not the rate at which we repeat ourselves. The second number is the one anyone actually wants and we cannot produce it.

The second number we had to throw away

We wanted to publish detection latency: how long a failure sat before anyone noticed. The ledger stores when a failure occurred and when the record was created, so the difference should give it.

It resolves on 52 of 4,308 records. On the rest the two dates land on the same day, which tells you nothing, and we record that as missing rather than as zero, because printing zero would invent a precision the data does not have.

Then we looked at the 52. Forty-three of them are negative.

A negative latency means the record was created before the failure it describes occurred. That is not possible, so what it actually means is that the two dates are not reliably ordered. Somebody typed one of them. The median across the measurable records is minus three days.

We are publishing no detection latency figure. Not a filtered one, not an absolute value, not a median of the nine positive records. A measure that is wrong in sign on 83 percent of the cases where it resolves at all is not a measure with some noise in it, it is a measure that does not work, and dressing it up would be the single most dishonest thing we could do with this dataset.

Why publish the holes

We publish what we measure, including the parts that do not flatter us. Two of the numbers we most wanted from this corpus are a recurrence rate we cannot fully trust and a latency figure we cannot report at all.

A study that only prints the figures that came out clean is not describing a real measurement process. It is describing the subset that survived, and leaving the reader to assume nothing else was attempted.

What we will do instead is say what has to change. Recurrence needs a link between records that share a cause, written by the person closing the second one, because no fingerprint will ever infer it. Latency needs the occurrence timestamp to come from the system rather than from a person, and until it does the field should not exist.

The full dataset, the method, and the complete list of what it cannot tell you are at the Agent Failure Index.