EVOHUNT: AI Auditing System Surpasses OpenAI's Codex Security

An open-source system outperforms OpenAI's Codex Security in vulnerability hunting

A team of researchers has developed an innovative system called EVOHUNT, which teaches AI agents to hunt for bugs in software by writing the audit method as plain text. Unlike traditional approaches that improve the AI model itself, EVOHUNT keeps the underlying model fixed and only refines an external "playbook" that guides the agent in its work.

The most significant result for security tool buyers is that an open-source model with an evolved playbook found real vulnerabilities at a higher rate than OpenAI's commercial product Codex Security, with a success rate of 11.3% compared to 9.2% across 371 test cases.

An innovative approach to learning

Most efforts to improve an AI audit agent replace the model with a larger one, update the operating software, and modify the workflow. EVOHUNT, on the other hand, separates these elements. The model and its operating software remain locked, while a text document improves autonomously.

The system operates in a three-agent loop: one examines the code and reports problems, a second verifies the results with known answers, and a third rewrites the playbook based on errors. The playbook starts empty and expands with each accepted change, with each version saved as code in a git repository. The two playbooks created this way reached approximately 1,600 and 2,200 lines of agent-written procedure.

Test results

Adding an evolved playbook to the closed GPT agent multiplied its working exploits sixfold. The open-source GLM agent achieved 11.3% and outperformed Codex Security in every measure monitored by the team.

From an economic standpoint, the teaching campaign lasted a month on subscription accounts for about $1,400. After this period, the actual audit is performed on cheaper open-source models. Applying an evolved playbook to a small Qwen model recovered most of the performance at about a third of the cost per case.

The system has already produced 28 confirmed vulnerability reports on 18 open-source projects, with a $1,500 bug bounty prize for a discovered bug.

Two distinct audit styles

Left to grow autonomously, the two playbooks developed opposite styles, reflecting a tension that every security team knows well.

The playbook grown with GPT became a precision tool. It limits the number of bug types it chases simultaneously and refuses to report anything without a working and reproduced exploit behind it. In a sample of its results, there were no false alarms.

The playbook grown with GLM went in the opposite direction and became an exhaustive snowplow. It repeats commands to keep searching more than thirty-five times and accepts subtler evidence to cover more ground. It captured more bugs overall and left more results for humans to examine. The choice between the two playbooks mirrors the daily decision that security teams must make between a short list to trust and a long list to triage.

Skill transfer without modifying weights

The part that gives the work its scope is the transfer. A playbook grown from a stronger teacher model made the weaker and cheaper models substantially better at the same job. In simple terms, the experience resides in a text file that any compatible model can pick up. An organization can pay for the expensive teaching pass once, then run the resulting playbook on cheap models for as long as desired.

Ziyue Wang, co-author of the paper, addressed the possibility of combining the two playbooks to achieve precision and coverage at the same time. "From an academic standpoint, our priority was to keep the experimental design clean," said Wang. "Even though we did not systematically map the exact adapter budget or mechanisms to resolve conflicting audit styles, our current results suggest that merging the precision of the GPT playbook with the breadth of the GLM playbook would improve the agent's overall vulnerability hunting capabilities."

Comparison with commercial products

The commercial benchmark, Codex Security, is a separate product, so the underlying model and operating software differ from the EVOHUNT runs. A cleaner test would pit an evolved playbook against one written by experts within the same agent. Wang stated that this type of baseline is hard to obtain. "Finding a perfectly controlled identical agent baseline is difficult because many high-level expert workflows are proprietary," said Wang, adding that the team wanted to test against Anthropic's Mythos but did not have access.

Wang links the design to a longer bet in AI research. "Our approach aligns with Rich Sutton's 'The Bitter Lesson,' the historical observation that general, scalable methods that leverage computation eventually outperform those that heavily depend on human-coded domain knowledge. Our work represents a step from '0 to 1' in this direction for security auditing."

Limitations and future considerations

Match rates hover in the single digits in most runs. Each playbook grew in a single training pass, so luck might explain part of the result. The agents are not fully comparable, as one runs on different operating software than the others. And the system that evaluates the results is itself an AI model. The authors flag all these points.

The bug count keeps growing. "Our approach has generated thousands of vulnerability results on open-source projects, including 28 zero-days confirmed by maintainers, and the number is growing," said Wang. Six more were confirmed after the paper was finalized.

Impact on the cybersecurity market

The emergence of EVOHUNT could revolutionize the cybersecurity market, offering a more affordable and effective alternative to existing commercial products. The use of open-source models could democratize access to advanced security audit tools, allowing even small and medium-sized businesses to implement robust security solutions without investing in expensive proprietary technologies.

Ethical and accountability considerations

The use of AI agents for vulnerability hunting raises important ethical questions. It is essential to ensure that these tools are used responsibly and not employed for malicious purposes. Additionally, transparency in the creation and use of playbooks is crucial for building trust among users and software developers.

Integration with other security technologies

EVOHUNT could be integrated with other existing security technologies, such as intrusion detection systems (IDS) and firewalls, to create a more comprehensive security ecosystem. This integration could enhance the ability to detect and respond to threats in real-time, providing more robust protection against cyberattacks.

Implications for software developers

For software developers, EVOHUNT represents a valuable tool for identifying and fixing vulnerabilities before they can be exploited. The use of AI agents for code auditing could accelerate the process of secure development, allowing developers to focus on creating innovative features rather than managing vulnerabilities.

Technical and implementation challenges

Despite the promising results, there are still significant technical challenges to address. The need for more controlled tests and access to proprietary expert workflows are major obstacles. Additionally, integrating different audit styles to achieve both precision and coverage is an area that requires further exploration.

Future prospects

With further developments and improvements, EVOHUNT could become an indispensable tool for security teams worldwide. The ability to teach cheap open-source models to perform complex security audits could revolutionize how vulnerabilities are identified and managed, enhancing software security globally.