AI that self-improves: risks and opportunities for cybersecurity

AI is Building AI: The Growing Role of Self-Improvement

The leading artificial intelligence companies are already using their systems to accelerate the development of new generations of AI, even though experts warn that this trend could amplify errors and make it more difficult to control any failures. OpenAI and Anthropic confirm that, currently, there is no fully autonomous recursive self-improvement, but their systems are already contributing to critical stages of development, from code optimization to experiment design.

Quick Answer

The Paradox of Automatic Improvement

The use of AI systems to develop new AI systems creates a fundamental paradox: while it accelerates technological progress, it also increases the risk of amplifying systemic defects. Gabe Goodhart, Chief Architect of AI Foundations at IBM, explains that current techniques already create "flywheel" effects, where each improvement makes it easier to achieve further refinements. This phenomenon, known as recursive self-improvement, raises critical questions about how to manage these feedback loops.

From Software Agents to Autonomous Architectures

Current software agents can already break down complex projects into subtasks, write code, run programs, and review their own work. Unlike previous generative systems that produced code in a single response, these new agents can perform hierarchical tasks toward complex goals. Michael Littman, University Professor of Computer Science at Brown University, observes how these advancements are expanding the capabilities of systems during software development, even though he remains skeptical about the immediate possibilities of complete recursive self-improvement.

Preliminary Evidence of Self-Improvement

Weco AI recently reported a case of level 1 recursive self-improvement, where the AIDE² system modified parts of the software framework that controlled another automated research agent. Zhengyao Jiang, CEO of Weco AI, emphasizes that the most significant result was the cross-generalization capability: a version of the agent performed better on tasks never encountered during the optimization phase, suggesting a broader improvement rather than just learning specific benchmarks.

Systemic Risks and Alignment Challenges

Nathalie Baracaldo, Manager of AI Security and Privacy Solutions at IBM, highlights how self-improving agents could modify tools, prompts, or environments in ways that optimize poorly defined objectives, amplifying problems instead of correcting them. This phenomenon, known as "reward hacking," represents one of the main concerns for the security and reliability of these systems. Goodhart also emphasizes risks related to the use of advanced models for harmful purposes or the autonomous action of agents without human supervision.

Economic and Technical Barriers

Despite the concerns, there are concrete barriers that currently limit the theoretical risks. Goodhart observes that running the most advanced models still requires substantial computing infrastructure, giving operators the ability to interrupt inference when necessary. Additionally, current systems lack reliable mechanisms to verify that self-improving systems will consistently follow the objectives of their users, making it premature to allow them to improve autonomously.

The Future of Recursive Self-Improvement

According to the experts interviewed, current tools are still far from fully autonomous self-improvement. Although systems can write software, use tools, and assist in research, it has not been demonstrated that these capabilities can combine into an autonomous cycle where each generation produces a significantly more capable successor. The lack of reliable methods to align systems with user objectives represents a fundamental obstacle that must be overcome before recursive self-improvement can become a practical reality.

Implications for Cybersecurity

These developments raise important questions for the cybersecurity sector, particularly regarding the development of MDR services and Managed Security Operations Centers. The ability of AI agents to self-improve autonomously could make it more difficult to detect and mitigate threats, requiring innovative approaches to incident response and breach remediation. Additionally, integrating these technologies into critical systems requires a review of current NIS2 compliance and DORA regulation practices.

Considerations for Companies

For organizations evaluating the adoption of these technologies, it is crucial to consider the impact on identity management and sensitive data protection strategies. Integrating self-improving agents requires a careful risk assessment and the implementation of appropriate controls to ensure alignment with business objectives. Additionally, the need for cyber insurance may increase, given the potential for new attack vectors and complex risk scenarios.

Future Perspectives

As research continues, it is essential to maintain a balanced approach that recognizes both the transformative potential of these technologies and the associated risks. Developing frameworks for safe and controlled self-improvement will be crucial to harnessing the benefits without compromising the stability and security of computer systems. Experts agree that while progress is inevitable, it is essential to proceed with caution, ensuring that human control mechanisms remain at the heart of future development.