Three out of four AI-generated patches leave something broken

Ask an advanced AI model to fix a real vulnerability and it will deliver something that looks like a solution. It reads like a patch that a maintainer would write. When there's a test, it often passes. About one time out of four, it is a solution.

The researchers at 1Password evaluated 6,080 patches for six recently disclosed CVEs, and the failures are rarely of the obvious type: an exploitation path blocked by a check with the vulnerable code still present, a bug fixed in one function and left untouched in its feature-for-feature twin, a memory error closed and a new one opened in the same helper.

Freenginx, a web server, allows site operators to write small Perl programs that run when a request arrives. A flaw in this feature, a use-after-free memory error, allows a visitor to crash the server. Trail of Bits sent the maintainers a fix via Patch the Planet, the initiative they run with OpenAI that delivers finished code to maintainers instead of another bug report. The maintainers rejected the version and wrote their own. The rejected version fixes two of the three points where the flaw resides; the maintainers' version fixes all three. Both also add a new way to crash the server, and reaching it requires only an ordinary client that initiates a request and then goes silent.

Off-by-1 Labs, the new security research group within 1Password, spotted that second crash and reported it to the freenginx maintainers on June 29; a fix came out on July 2. The same group then set up an automatic patcher on the original flaw and had ChatGPT 5.5 write 270 attempts at a fix. Its reviewer judged 114 of those attempts to have closed the original hole and found a new problem in each of the 114. Out of 270 attempts, nothing came out clean.

That campaign was a case study attached to the broader one, which pitted ChatGPT 5.5 and Claude Opus 4.8 against CVEs disclosed this spring and discarded every run in which the model went hunting for the official solution. About half of the surviving patches left at least one exploitable path open. Close to one in twenty introduced a new vulnerability, sometimes in addition to the failure to close the original. Most of the remaining misses closed the original bug but altered behavior along the way, rejecting input that the software used to accept or rewriting code.

The impact on the industry and future directions

This study raises important questions about the adoption of language models in the software industry. Many companies are already experimenting with using AI for automatic bug fixing, but the results of this study suggest that such practice could carry significant risks. New vulnerabilities introduced by AI-generated patches could be even more dangerous than the original vulnerabilities, as they might go unnoticed during traditional testing.

To mitigate these risks, it is crucial that developers adopt a critical approach toward AI-generated patches. This could include running more rigorous tests, manually analyzing the patches, and using advanced validation tools. Additionally, it might be helpful to develop specific methodologies for identifying and managing AI-introduced vulnerabilities.

The role of education and training

Effective use of language models for bug fixing requires a deep understanding of both the capabilities and limitations of these technologies. Developers must be trained not only in the practical use of these tools but also in the underlying principles and potential risks associated with them. This could include training courses on how to interpret the results of AI-generated patches, how to identify potential vulnerabilities, and how to integrate AI into existing development processes.

Furthermore, it is important that researchers continue to explore ways to improve the accuracy and security of AI-generated patches. This could include developing new algorithms, optimizing existing models, and creating frameworks for the validation and analysis of patches.

The implications for cybersecurity

The findings of this study have significant implications for the field of cybersecurity. If AI-generated patches are widely adopted without adequate understanding of their limitations, new vulnerabilities could emerge that might be exploited by malicious actors. This underscores the importance of a proactive approach to security, which includes not only the use of advanced tools but also a deep understanding of risks and potential threats.

Additionally, the results of this study could influence the development of standards and guidelines for the use of AI in bug fixing. Standardization bodies and expert groups may need to develop specific recommendations to ensure that AI-generated patches are safe and effective.

Conclusions

While modern language models offer great potential for automatic bug fixing, this study demonstrates that there are still many challenges to address. Developers must be aware of the limitations of these technologies and adopt a critical approach toward AI-generated patches. At the same time, researchers must continue to explore ways to improve the accuracy and security of these technologies.

In a world where cybersecurity is increasingly important, it is essential that the industry, researchers, and developers collaborate to ensure that emerging technologies are used safely and effectively. Only through a collaborative and proactive approach can we hope to address the complex challenges that lie ahead.