An AI agent built a security gap into our feature. Our process caught it before merge

From Gary Shaffer at TruStacks.

An AI agent built a feature for us recently and it had a security gap in it. A customer-written policy rule could read the environment of the process running it, which included an LLM API key, and it could make network calls. We caught it before merge. A separate AI agent that we'd given one narrow job found the gap, a second review of our fix found a route the first fix missed, and every finding became a test that had to fail before we'd accept the fix. Nothing reached production, no customer was exposed, and every test used an obviously fake secret.

I wanted to write this up because it's the same loop we sell, just run on our own engineering, and I think what we learned is more useful than "AI found a bug."

What we were building

TruStacks checks every change an agent proposes against signed policy. Customers can add their own rules on top of ours, and those rules can only tighten things, never loosen them. That already worked on self-hosted runners, but hosted customers had no way to deliver their rules. So the feature was to read rules from a rules/ folder in the customer's own platform repo, and rely on the branch protection and review they already trust for everything else.

An AI agent built it, with tests. Our automated pull request reviewer skipped this one because it had run out of review allowance, so nobody had actually reviewed the code yet, even though CI was green. We didn't merge on that. The feature changed what customer content our runner executes, which is a trust boundary, so we started a separate review agent and told it to attack the thing.

What the review found

1. 1Agent

   Built the feature

   Customer rules read from the platform repo, with tests. CI went green.

2. 2Review agent

   Attacked the trust boundary

   Found that a customer rule could read the runner's environment, including the LLM key, and make network calls.

3. 3Tests

   Fix 1, proven by a failing test

   The policy engine gets nothing in its environment but PATH; network and environment functions removed.

4. 4Review agent

   Reviewed the fix

   Found two schema functions that could still fetch a URL or read a local file.

5. 5Tests

   Fix 2: an allowlist, not a blocklist

   A reviewed list of allowed functions, and CI fails if anything new appears.

6. 6Person

   Asked where the gaps were

   Does the fix open a new gap somewhere else? Answered before anything merged.

Nothing reached production. Every test used an obviously fake secret.

The policy engine runs as a child process of our runner, and child processes inherit their parent's environment by default. That environment had the LLM API key in it. The engine also has a function that returns its own environment, so a few lines of customer rule could copy the key into a violation message, and those messages end up on the dashboard. There was a network-request function too, which meant a rule could send data out. The reviewer proved both of these using a fake secret.

So we fixed it. The policy engine now starts with nothing in its environment but PATH, and the network and environment functions are turned off using the engine's own capabilities mechanism. Customer rules get compiled against ours before they're admitted, so a rule that uses a forbidden function gets refused with a visible reason. The agents that need the LLM key still have it, and the policy engine, which never needed it, doesn't.

Then we sent a review agent at the fix, and it found something else. Two JSON-schema functions follow links inside a schema, and it used them to make a rule fetch a real web address and read a local file, with everything else locked down. With the environment stripped there wasn't a secret left to leak, but we had just promised no network and no files, and that promise was broken.

Why the second fix was different

Turning those two functions off was easy. The bigger change was how we decide what's allowed in the first place.

Blocklist

Fails open

Allow everything except the functions we listed.

Then

An upgrade adds a function nobody listed.

It is allowed, silently.

Allowlist

Fails closed

Allow only the functions a person reviewed.

Then

An upgrade adds a function nobody reviewed.

CI fails, and a person decides.

The first fix was a blocklist, meaning everything's allowed except the functions we named, and a function nobody thought of goes right through it. So now the list of allowed functions is a checked-in allowlist that a person reviewed, and CI fails if the engine ever allows something that isn't on it. If a future upgrade adds a new function, it can't quietly widen what a customer rule can do. The build stops and somebody has to make a decision.

The routine review came back later with three smaller findings. For example, a strict "signed bundles only" mode could still accept an unsigned bundle when signature checking was turned off. We fixed each of those the same way, with a test that failed first.

What I took from it

One review wasn't enough. The first review found the obvious route, we closed it, and a review of the fix found a less obvious one. If we'd stopped at the first green review, we would have shipped that second route.

Moving a trust boundary is a security change, even when nothing labeled security changed. The same weakness already existed for signed rule bundles, but using it there needed the customer's own signing key. The new feature lowered that bar to getting a pull request merged.

I also don't trust agents just because they're agents. We trusted their findings because each one became a failing test before anyone accepted a fix. The review I care about most is still the one a person does. Midway through the fix I asked, "Why would rules have secrets? And if we don't send the key to the runner, how are you going to get it and store it? Just making sure we're not creating gaps." I wasn't reading every line. I wanted to make sure the fix wasn't opening a new gap somewhere else, and I think that's where a person's time is best spent.

AI review didn't replace any of that, and it won't find everything. It found these because we layered the process on purpose. An agent proposes, independent agents attack it, tests decide, and a person approves.

Where to read the result

Hosted runners from 0.2.17 load customer rules from rules/ in the platform repo. The public reference lists exactly what a rule can and can't do, including the restricted functions and the stripped environment: Rules from your platform repo.

Agents propose. Policy decides. Humans approve.