Detecting infrastructure drift is easy and increasingly commoditised. Deciding what to do about it is the actual engineering problem.
Every team that turns on drift detection gets the same first week: dozens of findings, most of them noise, a few genuinely alarming. By week three the channel is muted. The tool worked exactly as advertised and changed nothing.
The gap is that drift is not one thing. Three different causes need three different responses, and treating them identically is why the alerts get ignored.
Classify before you automate
- Cloud-managed drift — fields the provider mutates on its own. Suppress these in code; they are not drift, they are a bad resource definition
- Emergency drift — someone fixed production at 3am. Legitimate, and the correct response is to backport it into code within the working week
- Unauthorised drift — a change nobody can account for. This is the one that matters, and it is usually a permissions problem rather than a Terraform problem
What we implement
A scheduled plan on a fixed cadence, with a non-empty diff raising a ticket that names the resource, the field and the last known author. Auto-remediation only on resources explicitly opted in — because automatically reverting a change made during an incident is how you extend the incident.
The long-term fix is upstream: if humans can change production directly, they will. Drift detection is a smoke alarm, not a sprinkler.