Rebuild instead of patching
Why I rebuilt the environment platform when the plan was only to fix it.
Context
Product environments shared Kubernetes namespaces, and a secrets migration was the next planned fix. Before starting it I ran a read-only audit of how every service was deployed.
Of 88 services across 133 repositories, 64 (73%) would silently undo that migration on their next deploy. The fix could be completed and then quietly reversed by ordinary releases.
Decision
Rebuild instead of patching. Each product environment moved to its own isolated cloud project on Cloud Run, behind one build-once, promote-by-tag pipeline, with configuration kept in git.
What we did
The first environment served 16 services in about four days, while testers were waiting for it. We then consolidated 15 repositories on the shared release pipeline and removed 29 legacy deployment workflows.
Production releases promote the exact commits that pre-production serves. A complete additional environment (26 services) was later built from Terraform through gated CI, which showed the design repeats.
Result
As of 15 November 2026 the estate is 5 isolated pre-production environments running 130 Cloud Run services and 25 worker pools. In one week in early November 2026 there were 19 successful production deploys and none failed.
Redesigning pipeline concurrency per environment took cancelled deployments from 79 in about 44 hours to 0 of the next 52. Between 2 September and 15 November 2026, 13 colleagues shipped changes through the same central pipeline.
Lesson
Audit before you fix. When most of the estate would undo the change, the change is the wrong unit of work, and the platform under it is the real problem.