EVMbench is a benchmark from OpenAI and Paradigm that measures how well AI agents detect, patch and exploit high-severity smart contract vulnerabilities. OpenAI says it draws on 117 curated vulnerabilities from 40 audits, and Paradigm says it uses real vulnerabilities from open code audits plus custom tasks, "containerized per-task so agents operate in realistic environments" (OpenAI: EVMbench; Paradigm: EVMbench).
​

This article explains what EVMbench tests, why AI smart contract security matters for agents, and how teams use AI auditing alongside human review.
​

EVMbench is an evaluation for AI agents working on Ethereum Virtual Machine smart contracts. It scores agents on three tasks: finding vulnerabilities in a contract repository, patching them, and exploiting them in a test environment. The goal is a repeatable way to compare models that claim to help with contract security.
​

In the detect task, agents audit a repository and are scored on how many of the known, ground-truth vulnerabilities they find (OpenAI). The EVMbench paper was accepted to ICML 2026 (OpenReview).
​

EVMbench tests AI agents by placing each one in an isolated environment with a real or custom contract codebase and asking it to detect, patch or exploit a known vulnerability. Results are compared against ground truth from prior audits, so a score reflects actual bugs found rather than plausible-sounding reports.
​

AI smart contract security matters because the same capability cuts both ways: models that find bugs for auditors can find them for attackers. A benchmark that measures exploitation gives defenders a sense of how capable automated attackers are becoming, and it gives auditors a baseline for which tools are worth adding to a review.
​

It also matters for coding agents. Developers increasingly use AI tools to write contract code, and generated code needs the same audit as human code. See smart contract audits for the human review process.
​

AI should not replace smart contract audits; current practice uses it as an additional pass alongside human auditors. Benchmarks like EVMbench measure recall on known vulnerability types, which is useful, but audits also cover design flaws, economic attacks and business logic that a benchmark task may not represent.
​

A related approach reduces how much custom contract code exists at all. Eco's Sauce SDK runs programs on an existing engine instead of deploying a new contract per program, so teams do not write, audit and deploy their own contract for each workflow (Sauce docs). That shifts review to the program's conditions. See security risks of autonomous AI agents.
​

AI tools for smart contract security look for the same vulnerability classes human auditors look for. EVMbench focuses on high-severity issues drawn from real audits, which usually fall into a handful of well-known categories. Knowing the categories helps teams judge where an AI pass is likely to help and where human review still carries the weight.
​

The right column is a general description of the problem's shape, not a benchmark result. For model-by-model results, read the EVMbench paper and announcement directly (OpenAI: EVMbench).
​

Teams that use AI in audits treat it as one more reviewer with a specific job, not as a replacement for the audit. A workflow that fits most projects:

EVMbench's three tasks map onto steps three, five and the patching that follows: detect, exploit to confirm, then patch (OpenAI).
​

A benchmark that measures whether an AI agent can exploit a contract tells defenders something uncomfortable but useful: how much skill an attacker can now automate. If agents get better at turning a known bug into a working exploit, the time between a bug being discoverable and being exploited shrinks. That raises the value of finding bugs before deployment and of limiting how much value any single contract can lose.
​

It also changes how teams should think about deployed code. Contracts that hold funds and cannot be paused or upgraded become more exposed as automated exploitation improves. Teams should know, for each contract, what happens if a bug is found tomorrow: can it be paused, who can do it, and how quickly.
​

Vibe coding means describing what you want to an AI tool and accepting the code it produces with little review. For web pages that may be fine. For smart contracts that hold money, the risks are specific:

AI coding tools are useful for contract work when the developer understands the code, writes the invariants, and treats generated code as a draft for review. Benchmarks like EVMbench measure the review side; they do not make generated code safe to deploy unreviewed.
​

One way to shrink the audit surface is to write less custom contract code in the first place. Many workflows, such as a swap with a minimum received, a payment to a list of recipients, or a rebalance to a target, do not need a new contract each time; they need a transaction that checks conditions and calls existing protocols.
​

Eco's Sauce SDK takes that approach. Programs are written in SauceScript, a subset of TypeScript syntax, compiled to bytecode for the V12 engine, and run through a Pot, with no new contract deployed per program (Sauce docs). The repository README puts it as "not a contract you write": there is no contract for the team to write, audit and deploy for each workflow (Sauce on GitHub). Review shifts to the program's conditions, which still need checking. The landing page's own wording is that a bytecode match shows reproducibility, not safety.
​

Smart contract security and agent security meet when an AI agent can call contracts. Even a well-audited contract can be misused by an agent that calls it with the wrong parameters or follows an injected instruction. The defenses are different layers: audits protect the contract from attackers; execution limits protect the agent's funds from the agent's own mistakes. See security risks of autonomous AI agents and when an AI agent sends crypto by mistake.
​

Choosing an AI tool for contract review is easier with a short, practical test than with marketing claims. Take a contract your team already audited, where the findings are known, and run the tool on the version before fixes. Then check four things.

Repeat the test when models change, since results can move between versions. Keep the test contract private so the tool cannot have seen the answers in public data.
​

Many large losses in crypto have come from outside contract code: stolen keys, compromised front ends, social engineering of signers, and misconfigured deployments. AI code review does not address those. A complete security program also covers key management, signer procedures, monitoring of deployed contracts, and an incident plan that says who can pause what. Benchmarks measure one slice of the problem well; teams need to cover the rest themselves.
​

AI-generated audit findings vary in quality, so reviewers should read them with a few questions in mind. Does the finding point to exact lines? Does it describe a concrete sequence of calls that triggers the problem? Does it state the impact in terms of who loses what? Can it be reproduced with a test? Findings that answer all four are worth a person's time immediately. Findings that answer none are often pattern matches that do not apply to this code, though they occasionally point at something real described badly.
​

OpenAI and Paradigm, per both organizations' announcement posts.
​

Agents' ability to detect, patch and exploit high-severity smart contract vulnerabilities.
​

No. AI tools find many known patterns but struggle with business logic and economic attacks that depend on the protocol's intent. Human audits remain standard.
​

You should not. AI-written contracts need the same review, testing and audit any contract needs. Treat generated code as a draft.
​

Detect asks the agent to find vulnerabilities, patch asks it to fix them, and exploit asks it to demonstrate an attack in a sandbox.
​

No. A clean AI review means the tool did not find issues it knows how to look for. Human auditors still review design, economics and the spec, which is where many serious and costly bugs live.
​

Sources: OpenAI, Paradigm and OpenReview pages linked inline, accessed November 2026.
​