Drift is a change that skipped the review

To keep staging and production consistent, close the side doors instead of diffing the results. Configuration drift is almost always a change that reached one environment by a path that skipped review: a console edit, a variable set by hand, a runtime patched in staging and never in production. Route every change to what runs through one reviewed pull request path, check it with the same rules on every target, and define production by a tier you declare rather than by what a cluster happens to be called. That stops the drift changes cause. It does not make staging's data or traffic look like production's, and this post says where that line is.

We found ours with a diff

We'll start with our own drift, because it's the clearest example we have.

TruStacks has a setting that turns on one of its delivery features: when a service's main branch moves, it proposes the new image tag as an ordinary pull request, so that even which version runs goes through policy and a person. On our hosted control plane, that setting had been switched on with a one-off command against the running deployment. It was never written into the deployment's configuration in git.

Nothing was wrong yet. The service behaved exactly as intended. But the next routine upgrade would have redeployed from the configuration in git, quietly switched the feature off, and nothing would have said so. A loop that never fires looks exactly like a loop with no work to do.

We caught it by diffing the live environment against the configuration we had declared, before an upgrade. That diff did its job. But look at what it found. The difference wasn't the problem; the difference was evidence of a path. A change had gone into production without passing through the place where every other change to that system is reviewed. The fix was not to correct the value. It was to put the value where the review is, so the next change to it has to come through the front door.

And yes: the setting that routes every image tag through a review was itself set without one. We don't think that's unusual. We think it's the normal shape of drift.

Every drift story has the same shape

Nathan Eddy's recent piece in Platform Engineering on environment consistency carries a story in exactly that shape. A Java runtime update was applied to staging during a security review and never applied to production. Every test passed. The first production login failed.

It is tempting to file that under "staging and production weren't identical." They weren't, but that's the outcome, not the cause. Two environments received one change by two different paths: one through whatever the security review used, the other through nothing at all. A config diff run that morning would have shown the version mismatch. It could not have shown that the runtime version was being managed outside the release process in the first place, and it would not have stopped the next one.

The general version: drift is what you get when there is more than one way to change what runs. The deploy pipeline is one way. The cloud console, a hand-run kubectl, a CI job with write access to deploy state, and a variable edited in a settings page are others. Each of those may be fast and convenient, and each one is a path the review never sees.

So the useful question is not "are my environments the same?" It is "how many ways are there to change them, and which of those ways are reviewed?"

One door

A reviewed pull request

✓Reviewed

Checked by the same rules on every target, and merged by a person.

Side doors

• The cloud console

  ✗The review never sees it

• A hand-run kubectl

  ✗The review never sees it

• A CI job with write access to deploy state

  ✗The review never sees it

• A variable edited in a settings page

  ✗The review never sees it

↓

What runs, in staging and in production

Every door leads here. A diff tells you the environments differ. It can’t tell you which path the difference took.

The useful question is how many ways there are to change what runs, and which of them are reviewed. TruStacks checks the changes its crew proposes. The side doors are governed by your own branch protection and access controls.

One door, for everything that decides what runs

TruStacks is AI delivery governance — the control layer between the agents that write your software and the systems that run it. For environment consistency, the property that matters is narrow and checkable: every change our crew makes to what runs arrives as a pull request against your platform repository, checked against the constitution before the pull request exists, and merged by a person. There is no other path out of the product.

"What runs" is broader than application code:

• Pipelines. CI is written as GitHub Actions workflows in your platform repository.
• Deployment manifests. Helm charts and ArgoCD Application manifests for Kubernetes targets. For ECS on Fargate, which is in Beta, a task definition and the GitHub Actions workflow that deploys it.
• The image tag. This is the one most setups exempt, because a tag bump feels too small to review. We went the other way. Your CI builds and pushes the image, but it is never given write access to the platform repository. When a service's main branch advances, TruStacks opens the new tag as an ordinary proposal: the same policy check, the same human merge. "Why is production running this commit?" is answered by a merged pull request with an approver on it, not by a CI log.

On Fargate the same idea runs a little further, because ECS has nothing like ArgoCD watching git. The deploy workflow the crew writes runs only when your platform repository's main branch changes, and it refuses to deploy any commit that isn't the current head of main, so re-running an old job can't quietly roll production back outside git. Which AWS account, region, roles, cluster and service a deploy targets live in a committed file rather than in repository variables, so pointing a deploy somewhere else is itself a reviewed change. Rolling back is a revert pull request.

"Prod" is a tier, not a name

The second source of inconsistency is subtler. Even when every change goes through review, the rules applied to it can quietly differ by environment, because "production" is decided by something incidental.

We had this bug. The constitution — the signed policy bundle we author and run at every tier — has a rule that an ArgoCD Application deploying to production must not sync automatically, so a person decides when production changes. Until recently, that rule recognised production by a folder name: an Application under a directory for a cluster called prod was protected. A production cluster called gke-us-east was not. Same intent, same review, and the rule silently did not apply.

The fix was to stop asking what the target is called. You declare each target with a tier, dev, uat or prod, and the rule keys on the tier. It is published as:

practice.argocd_prod_requires_manual_sync — ArgoCD Applications deploying to a prod-tier cluster must opt out of automated sync (spec.syncPolicy.automated absent).

A cluster literally named prod is still treated as production even if it was given a lower tier. When name and tier disagree, the rule takes the stricter reading.

practice.argocd_prod_requires_manual_sync

Cluster name

Declared tier

Before: matched a folder named prod

Now: keyed on the declared tier

Cluster name

prod

Declared tier

prod

Before

✓Manual sync required

Now

✓Manual sync required

Cluster name

gke-us-east

Declared tier

prod

Before

✗No gate

Now

✓Manual sync required

The bug. Same intent, same review, and the rule silently did not apply.

Cluster name

prod

Declared tier

dev

Before

✓Manual sync required

Now

✓Manual sync required

Name and tier disagree, so the rule takes the stricter reading.

Cluster name

Any other name

Declared tier

dev

Before

✗No gate

Now

✗No gate

The tier is yours to declare, and it has no default.

Only the second row changed, and it was the one that mattered. Declaring a target production is what turns the protection on, so a production target declared lower won’t get it unless it is literally named prod.

The same idea is what lets one constitution cover more than one kind of target. "A person approves promotion to production" is a single intent, and each target kind gets its own check for it. For a Fargate deploy it is:

practice.workflow_prod_deploy_requires_environment — On a prod-tier target, every job that deploys (aws-actions/amazon-ecs-deploy-task-definition) declares a GitHub environment. Presence only: the environment's reviewers are not verified.

We quote the second rule in full because its last sentence matters. The two checks are not equally strong, and saying they are would be exactly the kind of inconsistency this post is about:

• The ArgoCD check is enforcement in the artifact. A production Application without automated sync does not deploy until someone presses Sync. What the rule reads is what runs.
• The Fargate check is the presence of a reference. It proves the deploy job names a GitHub environment. Whether that environment actually requires a reviewer is a repository setting we don't read, and on private repositories it depends on your GitHub plan. On Fargate, the gate that always holds is the pull request review, because merging is what deploys.

Two more honest notes. The tier is yours to declare, and it has no default, because declaring a target production is what turns these protections on. A production target mislabelled dev won't get them (unless it is literally named prod). And both rules, Kubernetes and Fargate, live in one constitution you can read, test with stock OPA, and verify, in the public trustacks-policy repository.

An exception should be a commit

Eddy's piece also has a line about exceptions that we'd underline twice. It is Justin Beals of Strike Graph, quoted there, on an exception that has no end date:

"An exception without one becomes permanent architecture nobody remembers approving."

We agree, and we should be plain about where we stand: TruStacks has no waiver feature today. There is no way to grant an exception with an owner and an expiry inside the product. The constitution's rules are non-waivable by design, and we haven't built time-bound waivers for anything else.

What a single reviewed path does give you is the precondition. If the only way to change what runs is a pull request, then an exception can't be a setting someone flipped in a console. It has to be a diff, with an author, a reviewer, a timestamp and a reason in the description — a record of who approved it, which is the thing Beals says goes missing. The end date is still a convention your team has to keep. A commit doesn't expire on its own.

And the limit of "one door" deserves the same plainness. TruStacks checks the changes its crew proposes. A person with write access to your platform repository, or with console access to your cloud account, can still change things around it. Those are your side doors, and they're governed by your own branch protection and access controls, not by us. On Kubernetes, ArgoCD will at least show an Application as out of sync when the cluster no longer matches git. ECS has no equivalent: an out-of-band service update stays invisible until the next merge overwrites it. A read-only check for that is something we've written down as a follow-up, not something we ship.

Judge it by what breaks, not by what differs

Two environments can match line for line and still fail differently, and two that differ can both be healthy. The honest measure of consistency is what happens on delivery: incidents, rollbacks, regressions, and how often someone had to step in by hand. That is a better test than anything we could sell you.

We don't measure your incidents or your rollback rate. What a single reviewed path changes is how answerable those questions are. When something breaks in production, the first question is always what changed? If every change to what runs is a merged pull request, carrying the policy verdict and the name of the person who approved it, that question has an answer in your git history instead of in someone's memory. If the rollback is a revert, the rollback is in the same history. Counting manual interventions gets easier when there is one place a non-manual change could have come from.

What this doesn't fix

Two things, said plainly.

It doesn't reproduce production's conditions. Plenty of failures only appear under production's data shapes and traffic patterns, and test environments routinely have neither. Nothing in a pull request fixes that. A tier label doesn't make staging's data look like production's, and no policy rule can tell you that a query is fine at a thousand rows and falls over at ten million. That gap needs production-like data, load testing, and progressive rollouts. Policy is the wrong instrument for it, and we won't pretend otherwise.

It doesn't yet carry one change across your environments. Today a proposal is planned for one target at a time. Promoting a single reviewed change from staging to production as one tracked step, with each leg on the record, is on our roadmap and not built.

What it does fix is the most common way environments come apart: a change that went around the review. Make the review the only way in, apply the same rules to every target, and decide what production is by declaring it rather than by naming it.

Agents propose. Policy decides. Humans approve.

The rule texts above are copied from the published constitution in trustacks-policy. The staging runtime story and the quoted line about exceptions are from Nathan Eddy's "Platform Engineering's Next Test Is Environment Consistency", linked above. See the facts page for what is live versus roadmap, each entry with a verification date, and /solutions/gitops-delivery for the Kubernetes and Fargate rules side by side.