Multi-agent adversarial audit: when "all green" isn't enough

An implementation can compile cleanly, pass all its tests, and appear to satisfy every planned task. That is no guarantee it works end to end.

The hardest defects don’t always sit inside a particular function. They tend to surface in the seams of the system: a service nobody calls, a screen wired to the wrong method, a migration that doesn’t reflect the expected contract, or a claim of “verified” that was really only checked against a mock.

To investigate those places, StrayMark carries an engineering instrument we call a multi-agent adversarial audit, or MAAA.

What an MAAA is

An MAAA is a structured audit in which several AI models, drawn from different families, independently examine one unit of implementation: the Charter.

Every auditor gets the same working contract:

• what was meant to be built;
• which tasks were in scope;
• which files changed;
• which decisions and risks were recorded;
• which conditions had to hold before the work could close.

Their goal is not to confirm that the result “looks right”. It is to try to refute it with evidence.

The auditors inspect the code, trace flows between components, review tests, run whatever checks they are allowed to run, and compare the real implementation against the Charter’s obligations.

They do it in isolation. They can’t read each other’s reports and they can’t modify the project: they report what they find, they don’t fix it. That keeps a first opinion from contaminating the rest, and it lets us tell genuine agreement apart from mere repetition.

More than asking for several opinions

Running the same prompt four times is not a multi-agent audit. An MAAA introduces a separation of duties.

The independent auditors explore the same work from different angles. Some are especially strict about scope; others are better at following execution flows, real configuration, security, or technical debt.

The calibrator, or consolidator, reviews every report afterwards: it verifies the findings, removes duplicates, resolves contradictions, recalibrates severities, and can surface problems no auditor flagged. It is the same line we set out in Why StrayMark doesn’t automate unsupervised.

The accountable human keeps the final call: close the Charter, remediate it, or explicitly defer part of the work.

The result isn’t a vote among models. It’s an investigation with evidence, contradiction, and adjudication.

Enlarge figure

A real case: thousands of tests and a function that didn’t work

In a recent audit, the Charter under review implemented an emergency access or break-glass mechanism: a temporary, bounded, logged authorization for consulting sensitive information.

The solution looked solid. There were services, access controls, unit tests, a PostgreSQL integration, and thousands of passing tests.

The first round of the audit, however, found a decisive defect: the method that read the emergency authorization was implemented and tested, but not one of the application’s eight surfaces used it.

Each component worked on its own. The whole path did not. An operator could grant the authorization correctly and still reach no record at all.

The audit made it possible to locate the problem, remediate it, and add an integration test covering the real path: from the identity of whoever is asking to the record being read.

In the second round, four independent auditors confirmed the main defect was closed. But one of them found a problem from the same family: another public method, the one revoking the authorization, existed and had tests — and nothing in production called it either.

That’s the kind of failure a conventional review can miss: the capability is written, but it isn’t wired up.

The audit gets audited too

The most interesting part came afterwards.

After the first round, a guardian test had been written with a seemingly general rule: an authorization entry point must not exist without something calling it in production.

The consolidator found that the test only enumerated the method that had already failed. It didn’t cover the whole family of public capabilities.

The organization had learned a lesson, turned it into code, and still generalized it halfway. The detailed reports made that visible.

This is an important property of an MAAA: it doesn’t only produce findings about the software. It also generates information about:

• which models investigate more deeply;
• which tend to produce false positives;
• which are better at catching scope problems;
• which checks turned out to be insufficient;
• which new rules should enter the process;
• which defects got past everyone.

In other words, the audit becomes a system that can also examine and improve its own performance.

What the person running the project gets

For someone accountable for product, operations, or delivery, an MAAA helps separate ideas that are easily conflated.

Code produced and working functionality. A capability existing doesn’t mean the application can use it.

Passing tests and business criteria met. A suite can verify the components and skip the seam that connects them.

Deliberate debt and accidental omission. A documented risk has context, an owner, and a destination; a silent gap has none. It’s the distinction that holds up FollowUps.

Consensus and certainty. Several auditors agreeing raises confidence, but a lone finding can be the most important one of the round.

Activity and evidence. Recording that something was reviewed isn’t enough: it has to be clear what was checked, against which reality, and with what result.

That makes closing decisions more defensible, and leaves behind a technical memory worth reusing on the next piece of work.

An engineering capability of StrayMark

StrayMark doesn’t treat the audit as an isolated report at the end of a project. The Charter, the implementation logs, the Git range, the auditors’ reports, the consolidation, and the remediation actions all stay linked inside one governance flow.

An MAAA turns that whole into an instrument for projects where delivery needs more than speed:

• software with operational impact;
• systems handling sensitive information;
• implementations assisted by AI agents;
• teams that need traceability;
• organizations that can’t settle for “it seems to work”.

The promise isn’t that a group of models will find absolutely every defect. It’s that every claim that matters can be challenged, every finding can be traced, and every failure of the process itself can turn into a permanent engineering improvement.

With StrayMark, AI doesn’t only help build software. It also takes part in a system designed to test what it built, challenge its own conclusions, and leave the final call where it belongs: in human hands. It’s the same bet we described in An audit that keeps you in the room.

---

About StrayMark

StrayMark is a framework and command-line tool, written in Rust, for the discipline that working with AI agents demands: it externalizes scope, decisions, and risks into versioned files next to the code, so the agent executes against constraints instead of inventing its own. The MAAA is one of its capabilities. It’s free and open source under the MIT license, and its documentation lives at straymark.dev.