← All decisions
Decision 03 · case study

Refuse it twice

Observability from zero, audited before sign-off.

Context

I commissioned an internal monitoring product: 13 screens, backed by 4,146 tests. It looked finished and every test passed.

Decision

I rejected its first two versions. For the third I ordered two independent audits of the live build, comparing what each screen showed with what the running systems reported.

What we did

The audits found 30 of 78 displayed fields wrong. All were fixed before sign-off.

The same standard applied to the rest of the observability work. An audit of 104 services found that the main gap in trace-to-log correlation was one missing deploy-time setting. On the two services measured, linked log lines went from 0% to 100%, and the fix was rolled out to 17 service families.

Result

Pre-production now has in-region monitoring for five environments: dashboards behind single sign-on, 27 alert policies, 5 service-level objectives and 69 of 69 synthetic checks green. Vendor-health monitoring is self-hosted in the same region for data residency, with 143 checks.

Lesson

A passing test suite is not evidence that the screen is right. Done means read back from the live system, and someone other than the author has to do the reading.

Cairo ↔ Riyadh · updated November 2026