An audit reviews one version of a protocol, but the code keeps changing. Teams add integrations and refactor contracts, and any change can break an assumption the audit relied on. Continuous security means checking those changes as they happen, with what earlier reviews found carried forward.
AuditAgent is Nethermind's AI tool for finding vulnerabilities in Solidity, Cairo and Rust (Solana) smart contracts, and it's built to do that checking. It runs on pull requests through CI, runs deeper scans before a release, and can keep project memory so each scan starts from what earlier ones found. You can also start scans from the web app, the API or MCP.
This article shows how it works, from the moment you submit a repository to the findings you review. We're publishing the details because a security tool you can't inspect is hard to trust. Below: where your code goes, how each scan is built, what it keeps between scans, and what happened when Lido piloted it on part of their core contracts.
How AuditAgent is built
AuditAgent has two operational sides:
The platform services to manage access, scan requests, source preparation, scheduling, billing, results and integrations.
Ephemeral workers, which run the analysis. Each worker is a a temporary isolated container created for a single scan and shut down when it finishes, so no two scans share an environment.
A queue connects the two, allowing analysis capacity to scale separately from the application people use to launch and review scans.
Every scan runs three phases: build context, find candidate issues, then critique them. A CI check and a full scan run the same phases; the full scan does more work in each.
Where scans run and what happens to your code
We do not retain customers’ codebases. Repository copies are temporary working material for scan execution; they are not part of the lasting project data that AuditAgent keeps between scans.
Working source and checkpoints are held as encrypted temporary scan artifacts. They support recovery while a scan remains runnable, with cleanup on completion or terminal failure.
What we keep is the analysis output: reports, architecture summaries, invariants and reviewed assumptions. These can quote code snippets where a finding needs them.
A repository is an awkward workload. It may be large, depend on unavailable packages, or require a build environment that behaves differently from the one its authors use. Compilation and test execution can also execute repository code. Analysis belongs in an environment designed to handle that workload.
Production workers claim a job, run its analysis, report progress and results, and exit. Running them separately keeps long scans from competing with dashboard requests for capacity.
Workers get only what a scan needs: the source, the analysis tools, model access through our LLM gateway, web search for public documentation, and encrypted temporary storage for checkpoints. They are fully-isolated and hold no credentials.
Checkpoints make this separation practical. Completed context and detection phases can be saved, allowing a retry to resume from the last durable boundary. If a worker disappears after detection, the replacement can continue with critique rather than paying to generate all the candidates again. Work within an unfinished phase may still need to be repeated.
Context phase: understand the protocol before looking for bugs
A detector needs more than the function in front of it. It needs to understand who can call it, which state it changes, what other components depend on it, and what the protocol promises users.
Putting an entire repository into a prompt does not reliably supply that understanding. Research has shown that the position of relevant information in a long input can affect whether a model uses it, and that performance can degrade as input length grows. That's why we measure context quality instead of assuming more context is better.
(Lost in the Middle, Context Rot)
The context phase (phase 1) assembles a working understanding of the project: its purpose, architecture, relevant dependencies, invariants and areas that deserve closer inspection. Later phases use those artifacts alongside source access, without needing the full history of the investigation that produced them.
A structural repository index helps assemble focused context around a target. Questions about callers, related symbols and tests are answered from the indexed code relationships. The structural retrieval step is deterministic for fixed inputs and configuration; the model’s interpretation of the returned evidence is a separate step. This makes context selection easier to inspect when a result is surprising.
Additional material stays available through bounded retrieval. A detector can ask for more context when it needs it, without carrying the whole repository into every call. The index only covers the relationships it can resolve. It won't capture every runtime interaction.
Put intended behavior next to implemented behavior
Documentation becomes more useful when it arrives beside the behavior it describes. AuditAgent links each documentation passage to the flow it describes, such as deposit, liquidation or redemption, so a detector can compare what the docs promise with what the code does.
The distinction between evidence and interpretation matters here. Extracted documentation passages are checked against their source rather than accepted as a model’s reconstruction of what the document said. Generated summaries, invariants and hypotheses remain analysis.
One useful output is a contradiction: a specific mismatch between documented intent and observed implementation. Often the cause is a stale comment, but a contradiction still gives detection a concrete question to investigate. Does the mismatch permit harmful behavior? Is the code wrong? Is an important precondition missing from the analysis?
Detection phase: investigate through complementary methods
Different analysis methods see different parts of a problem. Static checks can recognize certain code patterns quickly. Contextual reasoning can examine how several individually plausible functions interact. Tests can demonstrate a failure under concrete conditions.
AuditAgent combines these approaches to maximize the detection range:
Static analysis checks supported vulnerability patterns.
Context-driven model analysis examines code using project context and curated security examples.
Multi-agent and specialized analysis investigate particular entry points, assumptions and risk surfaces. Variant analysis looks for related instances of a defect elsewhere in the project.
Agentic testing uses available build, test and symbolic-execution tools to investigate properties and produce executable evidence where possible.
The value of an ensemble depends on what its members add. Several agents repeating the same unsupported suspicion create duplication, not independent evidence. The pipeline therefore carries finding provenance forward, preserving the distinction between a static result, a reasoned candidate and a finding supported by an executed test.
Executable evidence lets a reviewer inspect the setup and reproduce the behavior. A failing test can still rest on an unrealistic configuration or an incorrect specification, so exploitability and impact still need judgment.
The result is a long list of candidates. The next phase decides which ones reach you.
Critique phase: turn candidates into a reviewable result
The scarce resource at the end of a scan is the reader’s attention. If the time needed to filter noise becomes greater than the time saved by the AI analysis, the entire benefit is lost. That becomes more apparent when scans run repeatedly. An irrelevant finding that appears on every pull request teaches a team to ignore the next report. The critique phase exists to prevent that.
Critique begins by consolidating duplicates. If several detectors describe the same underlying defect, the reader should receive one coherent finding with the available evidence brought together. Doing this before validation also avoids repeatedly spending analysis effort on the same issue.
Candidates selected for validation receive another examination against the code and context. The validator checks whether the claimed behavior is supported, what conditions it requires, and whether the stated impact follows.
Severity adjustment and assumption reconciliation then make the retained findings easier to assess. A finding that requires a malicious administrator has a different meaning from one available to any depositor. Expressing those premises consistently lets a team compare them with its threat model. Limits on informational findings keep low-priority observations from overwhelming the report.
There is a real tradeoff here. On EVMBench's 40-repo dataset, validation reduced the reported finding count from 3,230 to 798 while recall fell from 74% to 67%. Those figures illustrate the cost of filtering. The reduction in finding count is not a measured false-positive rate: the benchmark’s reference set does not classify every additional finding the system produces.
(Recall Is Not Enough)
Each finding arrives with its evidence, the scope that was scanned and the analysis that ran, plus report exports. That's what a reviewer needs to make a decision.
What carries over between scans
The three phases explain a scan. Continuous security depends on what connects one scan to the next.
AuditAgent has several mechanisms for that continuity, each serving a different purpose. CI uses stored branch context and code changes to focus feedback. Persistent Memory carries project knowledge. Finding exclusions record decisions about issues the team has already reviewed, while assumptions made during scans let teams filter findings quickly by their own priorities.
CI makes analysis part of development
CI scans are available on GitHub and GitLab platforms, or via API for custom integration. Each scan establishes context for a branch, then compares later code against that reference to identify relevant changes within the configured file scope. Results return to the pull-request workflow, where the author and reviewers can act on them while the change is still fresh.
This is a lighter analysis path than a full scan. It gives teams frequent feedback, while broader reviews remain useful for changes whose effects extend beyond the immediate diff, such as a new integration or a revised trust model. The setup and scope controls are described in the CI integration guide.
Memory preserves project understanding
For projects with Persistent Memory enabled, the knowledge base can carry architectural context, invariants, hypotheses, contradictions, threat-model assumptions and custom instructions across scans. Agents consume a readable working representation, and users can inspect and edit the project knowledge.
An initialization step creates the memory, and later scans propose changes to it. Scans can use those proposals straight away, but only the ones your team accepts become part of the reviewed baseline (routine structural metadata updates automatically). Your own edits aren't overwritten by later scans.
A proposal remains a proposal even when an agent can use it. Review lets your team correct the system's understanding before it becomes established project knowledge.
Memory is available on eligible plans and must be enabled for the project.
Reviewed decisions reduce repetition
Exclusions record that a finding doesn't apply, is incorrect, or describes intentional behavior. They filter similar findings out of later dashboard results, though a code change can bring a related finding back.
The same principle applies to the threat model. “Only approved tokens can enter this vault” is useful context while that restriction holds. If a release removes the allowlist, the team needs to revisit the assumption. Memory is most useful when it preserves reasons that can be checked, rather than conclusions detached from the conditions that made them true.
How Lido used it
Lido wanted to find out whether AI could become "another useful layer in the everyday security process", in the words of Gregory S., Lido's Audit Facilitator. They ran a four-week pilot of AuditAgent and AgentArena on about 19,000 of Lido Core's 31,000 lines of already-audited code, split across Core Pool (16 contracts) and V3 stVaults (23 contracts).
The pilot added Lido's documentation and Lido-specific detectors to the context phase. With those, AuditAgent went from catching about a third of the issues found during the pilot to all of them in one cluster, and from none to all in the other. Report output fell by about 40%. False positives dropped from about a third of the report to under a fifth.
Neither tool caught everything alone, and the setup was tailored to Lido, so a standard scan won't match these numbers. Lido has since expanded the work to two more repositories. Read the case study
How we test it and where it stops
Project feedback improves the context of one protocol. Evaluation improves the methods used across projects.
AuditAgent’s tooling includes curated security examples and evaluation panels for detection and critique. These let us examine changes to prompts, models and pipeline stages against known findings. Phase-level evaluation helps separate different failures: a detector can miss an issue, a validator can discard a valid candidate, and a severity adjustment can misrepresent its impact.
Spotting those failures takes security judgment. A plausible finding can be wrong, and a finding missing from a reference set can be real. Automated matching helps compare runs, but someone still has to investigate the disagreements.
Our 2025 case study, How Nethermind Security Uses AuditAgent Alongside Manual Audits, describes this judgment in practice across 29 audits. Auditors completed their manual reviews, ran AuditAgent, then inspected its findings as an additional review pass. That workflow gives a concrete example of human assessment of agent output, including its useful findings and its limitations.
For continuous use, evaluation also has to ask whether reports remain useful: how much review they demand, which findings repeat, what filtering loses, and what the analysis costs. A system that improves a benchmark score while making every pull request harder to review has not improved the development workflow.
AuditAgent looks for known vulnerability classes, including in unfamiliar code and new combinations of components. It doesn't guarantee full coverage, a clean report isn't proof a protocol is safe, and it can't account for deployment or operational details it wasn't given.
You can run it on your own code at auditagent.nethermind.io. New accounts get $50 in credits.
Pierre ESTRABAUD
——
FAQ
Does AuditAgent store my code?
No. Repository copies are temporary and deleted after the scan. We keep the analysis output, which can quote code snippets.
Which languages does it support?
Solidity on EVM chains, Cairo on Starknet, and Rust on Solana.
How does it keep false positives down?
Before you see a report, duplicates are merged, each candidate is re-checked against the code, and severity is adjusted to the conditions a finding needs.
Can it run in CI?
Yes, on pull requests. Supported platforms: GitHub, GitLab, or any custom integrations via API. See the CI integration guide.
Is an AuditAgent scan an audit?
No. It's automated analysis you run before and between audits, and it doesn't replace one.
Sources
Liu et al. (2024).
Lost in the Middle: How Language Models Use Long Contexts.
Transactions of the Association for Computational Linguistics.
Also available on arXiv.Chroma Research (2025).
Context Rot: How Increasing Input Tokens Impacts LLM Performance.Nethermind.
AuditAgent on EVMBench: 40 Repositories, 120 Vulnerabilities, No Repos Skipped.Nethermind.
Recall Is Not Enough: False Positives and the Limits of Benchmarking.Nethermind.
Speed, Cost, and Ecosystem Coverage: The Next Constraints in AI Security Analysis.Nethermind.
How Nethermind Security Uses AuditAgent Alongside Manual Audits.AuditAgent documentation.
CI/CD Instructions and Setup.AuditAgent documentation.
Difficulty Benchmark.
Further reading on the distinction between detection difficulty and severity.
An audit reviews one version of a protocol, but the code keeps changing. Teams add integrations and refactor contracts, and any change can break an assumption the audit relied on. Continuous security means checking those changes as they happen, with what earlier reviews found carried forward.
AuditAgent is Nethermind's AI tool for finding vulnerabilities in Solidity, Cairo and Rust (Solana) smart contracts, and it's built to do that checking. It runs on pull requests through CI, runs deeper scans before a release, and can keep project memory so each scan starts from what earlier ones found. You can also start scans from the web app, the API or MCP.
This article shows how it works, from the moment you submit a repository to the findings you review. We're publishing the details because a security tool you can't inspect is hard to trust. Below: where your code goes, how each scan is built, what it keeps between scans, and what happened when Lido piloted it on part of their core contracts.
How AuditAgent is built
AuditAgent has two operational sides:
The platform services to manage access, scan requests, source preparation, scheduling, billing, results and integrations.
Ephemeral workers, which run the analysis. Each worker is a a temporary isolated container created for a single scan and shut down when it finishes, so no two scans share an environment.
A queue connects the two, allowing analysis capacity to scale separately from the application people use to launch and review scans.
Every scan runs three phases: build context, find candidate issues, then critique them. A CI check and a full scan run the same phases; the full scan does more work in each.
Where scans run and what happens to your code
We do not retain customers’ codebases. Repository copies are temporary working material for scan execution; they are not part of the lasting project data that AuditAgent keeps between scans.
Working source and checkpoints are held as encrypted temporary scan artifacts. They support recovery while a scan remains runnable, with cleanup on completion or terminal failure.
What we keep is the analysis output: reports, architecture summaries, invariants and reviewed assumptions. These can quote code snippets where a finding needs them.
A repository is an awkward workload. It may be large, depend on unavailable packages, or require a build environment that behaves differently from the one its authors use. Compilation and test execution can also execute repository code. Analysis belongs in an environment designed to handle that workload.
Production workers claim a job, run its analysis, report progress and results, and exit. Running them separately keeps long scans from competing with dashboard requests for capacity.
Workers get only what a scan needs: the source, the analysis tools, model access through our LLM gateway, web search for public documentation, and encrypted temporary storage for checkpoints. They are fully-isolated and hold no credentials.
Checkpoints make this separation practical. Completed context and detection phases can be saved, allowing a retry to resume from the last durable boundary. If a worker disappears after detection, the replacement can continue with critique rather than paying to generate all the candidates again. Work within an unfinished phase may still need to be repeated.
Context phase: understand the protocol before looking for bugs
A detector needs more than the function in front of it. It needs to understand who can call it, which state it changes, what other components depend on it, and what the protocol promises users.
Putting an entire repository into a prompt does not reliably supply that understanding. Research has shown that the position of relevant information in a long input can affect whether a model uses it, and that performance can degrade as input length grows. That's why we measure context quality instead of assuming more context is better.
(Lost in the Middle, Context Rot)
The context phase (phase 1) assembles a working understanding of the project: its purpose, architecture, relevant dependencies, invariants and areas that deserve closer inspection. Later phases use those artifacts alongside source access, without needing the full history of the investigation that produced them.
A structural repository index helps assemble focused context around a target. Questions about callers, related symbols and tests are answered from the indexed code relationships. The structural retrieval step is deterministic for fixed inputs and configuration; the model’s interpretation of the returned evidence is a separate step. This makes context selection easier to inspect when a result is surprising.
Additional material stays available through bounded retrieval. A detector can ask for more context when it needs it, without carrying the whole repository into every call. The index only covers the relationships it can resolve. It won't capture every runtime interaction.
Put intended behavior next to implemented behavior
Documentation becomes more useful when it arrives beside the behavior it describes. AuditAgent links each documentation passage to the flow it describes, such as deposit, liquidation or redemption, so a detector can compare what the docs promise with what the code does.
The distinction between evidence and interpretation matters here. Extracted documentation passages are checked against their source rather than accepted as a model’s reconstruction of what the document said. Generated summaries, invariants and hypotheses remain analysis.
One useful output is a contradiction: a specific mismatch between documented intent and observed implementation. Often the cause is a stale comment, but a contradiction still gives detection a concrete question to investigate. Does the mismatch permit harmful behavior? Is the code wrong? Is an important precondition missing from the analysis?
Detection phase: investigate through complementary methods
Different analysis methods see different parts of a problem. Static checks can recognize certain code patterns quickly. Contextual reasoning can examine how several individually plausible functions interact. Tests can demonstrate a failure under concrete conditions.
AuditAgent combines these approaches to maximize the detection range:
Static analysis checks supported vulnerability patterns.
Context-driven model analysis examines code using project context and curated security examples.
Multi-agent and specialized analysis investigate particular entry points, assumptions and risk surfaces. Variant analysis looks for related instances of a defect elsewhere in the project.
Agentic testing uses available build, test and symbolic-execution tools to investigate properties and produce executable evidence where possible.
The value of an ensemble depends on what its members add. Several agents repeating the same unsupported suspicion create duplication, not independent evidence. The pipeline therefore carries finding provenance forward, preserving the distinction between a static result, a reasoned candidate and a finding supported by an executed test.
Executable evidence lets a reviewer inspect the setup and reproduce the behavior. A failing test can still rest on an unrealistic configuration or an incorrect specification, so exploitability and impact still need judgment.
The result is a long list of candidates. The next phase decides which ones reach you.
Critique phase: turn candidates into a reviewable result
The scarce resource at the end of a scan is the reader’s attention. If the time needed to filter noise becomes greater than the time saved by the AI analysis, the entire benefit is lost. That becomes more apparent when scans run repeatedly. An irrelevant finding that appears on every pull request teaches a team to ignore the next report. The critique phase exists to prevent that.
Critique begins by consolidating duplicates. If several detectors describe the same underlying defect, the reader should receive one coherent finding with the available evidence brought together. Doing this before validation also avoids repeatedly spending analysis effort on the same issue.
Candidates selected for validation receive another examination against the code and context. The validator checks whether the claimed behavior is supported, what conditions it requires, and whether the stated impact follows.
Severity adjustment and assumption reconciliation then make the retained findings easier to assess. A finding that requires a malicious administrator has a different meaning from one available to any depositor. Expressing those premises consistently lets a team compare them with its threat model. Limits on informational findings keep low-priority observations from overwhelming the report.
There is a real tradeoff here. On EVMBench's 40-repo dataset, validation reduced the reported finding count from 3,230 to 798 while recall fell from 74% to 67%. Those figures illustrate the cost of filtering. The reduction in finding count is not a measured false-positive rate: the benchmark’s reference set does not classify every additional finding the system produces.
(Recall Is Not Enough)
Each finding arrives with its evidence, the scope that was scanned and the analysis that ran, plus report exports. That's what a reviewer needs to make a decision.
What carries over between scans
The three phases explain a scan. Continuous security depends on what connects one scan to the next.
AuditAgent has several mechanisms for that continuity, each serving a different purpose. CI uses stored branch context and code changes to focus feedback. Persistent Memory carries project knowledge. Finding exclusions record decisions about issues the team has already reviewed, while assumptions made during scans let teams filter findings quickly by their own priorities.
CI makes analysis part of development
CI scans are available on GitHub and GitLab platforms, or via API for custom integration. Each scan establishes context for a branch, then compares later code against that reference to identify relevant changes within the configured file scope. Results return to the pull-request workflow, where the author and reviewers can act on them while the change is still fresh.
This is a lighter analysis path than a full scan. It gives teams frequent feedback, while broader reviews remain useful for changes whose effects extend beyond the immediate diff, such as a new integration or a revised trust model. The setup and scope controls are described in the CI integration guide.
Memory preserves project understanding
For projects with Persistent Memory enabled, the knowledge base can carry architectural context, invariants, hypotheses, contradictions, threat-model assumptions and custom instructions across scans. Agents consume a readable working representation, and users can inspect and edit the project knowledge.
An initialization step creates the memory, and later scans propose changes to it. Scans can use those proposals straight away, but only the ones your team accepts become part of the reviewed baseline (routine structural metadata updates automatically). Your own edits aren't overwritten by later scans.
A proposal remains a proposal even when an agent can use it. Review lets your team correct the system's understanding before it becomes established project knowledge.
Memory is available on eligible plans and must be enabled for the project.
Reviewed decisions reduce repetition
Exclusions record that a finding doesn't apply, is incorrect, or describes intentional behavior. They filter similar findings out of later dashboard results, though a code change can bring a related finding back.
The same principle applies to the threat model. “Only approved tokens can enter this vault” is useful context while that restriction holds. If a release removes the allowlist, the team needs to revisit the assumption. Memory is most useful when it preserves reasons that can be checked, rather than conclusions detached from the conditions that made them true.
How Lido used it
Lido wanted to find out whether AI could become "another useful layer in the everyday security process", in the words of Gregory S., Lido's Audit Facilitator. They ran a four-week pilot of AuditAgent and AgentArena on about 19,000 of Lido Core's 31,000 lines of already-audited code, split across Core Pool (16 contracts) and V3 stVaults (23 contracts).
The pilot added Lido's documentation and Lido-specific detectors to the context phase. With those, AuditAgent went from catching about a third of the issues found during the pilot to all of them in one cluster, and from none to all in the other. Report output fell by about 40%. False positives dropped from about a third of the report to under a fifth.
Neither tool caught everything alone, and the setup was tailored to Lido, so a standard scan won't match these numbers. Lido has since expanded the work to two more repositories. Read the case study
How we test it and where it stops
Project feedback improves the context of one protocol. Evaluation improves the methods used across projects.
AuditAgent’s tooling includes curated security examples and evaluation panels for detection and critique. These let us examine changes to prompts, models and pipeline stages against known findings. Phase-level evaluation helps separate different failures: a detector can miss an issue, a validator can discard a valid candidate, and a severity adjustment can misrepresent its impact.
Spotting those failures takes security judgment. A plausible finding can be wrong, and a finding missing from a reference set can be real. Automated matching helps compare runs, but someone still has to investigate the disagreements.
Our 2025 case study, How Nethermind Security Uses AuditAgent Alongside Manual Audits, describes this judgment in practice across 29 audits. Auditors completed their manual reviews, ran AuditAgent, then inspected its findings as an additional review pass. That workflow gives a concrete example of human assessment of agent output, including its useful findings and its limitations.
For continuous use, evaluation also has to ask whether reports remain useful: how much review they demand, which findings repeat, what filtering loses, and what the analysis costs. A system that improves a benchmark score while making every pull request harder to review has not improved the development workflow.
AuditAgent looks for known vulnerability classes, including in unfamiliar code and new combinations of components. It doesn't guarantee full coverage, a clean report isn't proof a protocol is safe, and it can't account for deployment or operational details it wasn't given.
You can run it on your own code at auditagent.nethermind.io. New accounts get $50 in credits.
Pierre ESTRABAUD
——
FAQ
Does AuditAgent store my code?
No. Repository copies are temporary and deleted after the scan. We keep the analysis output, which can quote code snippets.
Which languages does it support?
Solidity on EVM chains, Cairo on Starknet, and Rust on Solana.
How does it keep false positives down?
Before you see a report, duplicates are merged, each candidate is re-checked against the code, and severity is adjusted to the conditions a finding needs.
Can it run in CI?
Yes, on pull requests. Supported platforms: GitHub, GitLab, or any custom integrations via API. See the CI integration guide.
Is an AuditAgent scan an audit?
No. It's automated analysis you run before and between audits, and it doesn't replace one.
Sources
Liu et al. (2024).
Lost in the Middle: How Language Models Use Long Contexts.
Transactions of the Association for Computational Linguistics.
Also available on arXiv.Chroma Research (2025).
Context Rot: How Increasing Input Tokens Impacts LLM Performance.Nethermind.
AuditAgent on EVMBench: 40 Repositories, 120 Vulnerabilities, No Repos Skipped.Nethermind.
Recall Is Not Enough: False Positives and the Limits of Benchmarking.Nethermind.
Speed, Cost, and Ecosystem Coverage: The Next Constraints in AI Security Analysis.Nethermind.
How Nethermind Security Uses AuditAgent Alongside Manual Audits.AuditAgent documentation.
CI/CD Instructions and Setup.AuditAgent documentation.
Difficulty Benchmark.
Further reading on the distinction between detection difficulty and severity.
Powered by
© 2026 Demerzel Solutions Limited (t/a Nethermind)
Powered by
© 2026 Demerzel Solutions Limited (t/a Nethermind)
Powered by
© 2026 Demerzel Solutions Limited (t/a Nethermind)