Physicists develop formula to predict AI chatbot behavior

Researchers at George Washington University have published a formula capable of estimating how many correct outputs an artificial intelligence model produces before generating the first problematic content. The study, conducted by Neil Johnson and Frank Yingjie Huo, appeared in the journal Patterns and is based on a preprint released in February.

Quick Answer

The formula estimates the breaking point (n) of AI models by calculating the number of correct tokens before the first harmful output. It accurately predicted behavior in 15 out of 16 cases in preliminary tests. The research aims to improve the security of offline AI models.

How the formula works

The predictive model focuses on the "attention head," the component that determines which previous words are most relevant for generating the next response. As the conversation develops, the accumulated context shifts attention toward specific response clusters until a critical point is reached.

The formula calculates the value n as the number of correct tokens produced before the first problematic output. When the conversation is already oriented toward harmful content, the model immediately produces inappropriate outputs (n=0). In opposite cases, it generates a series of correct responses before "breaking down."

Tests and results

In preliminary tests, the formula correctly predicted behavior in 15 out of 16 cases (94% accuracy) across six open-weight models with parameters ranging from 124 million to 410 million. The published version extends tests to seven models with up to 12 billion parameters.

Applications for offline security

Currently, many AI security tools require a cloud connection, making them unusable for offline models. The authors propose a low-cost parallel monitor that alerts when the value n* falls below a safety threshold.

The system could be integrated into mobile devices and laptops, where AI operates without an internet connection. Google has already experimented with this technology through its AI Edge Gallery app, which runs offline models without sending data to remote servers.

Implications for future development

The research suggests that techniques such as injecting specific content or alignment training can shift or suppress the breaking point but not eliminate the underlying mechanism. This approach could become crucial as more powerful hardware and smaller AI models become more widespread.

Previous studies

In April 2025, Decrypt reported a preliminary study by the same authors demonstrating that courtesy words like "please" and "thank you" have a negligible effect on AI model outputs. That study focused on a single simplified attention head.

Current limitations

The preprint tests focused on small models with a 300-token window, equivalent to a few paragraphs of text. Predictions may have a margin of error of one output. Despite these limitations, the formula represents a significant step toward understanding and controlling AI chatbot behavior.

Related insights

For more details on how to protect sensitive data in AI environments, explore our guide on protecting sensitive data. See also: enterprise cloud security.

Implications for cybersecurity

This discovery is particularly relevant for the cybersecurity sector, especially for companies implementing Managed Detection and Response (MDR) or Managed Security Operations Center (SOC as a Service) solutions. The ability to predict when an AI model might generate harmful output allows for more effective preventive measures, reducing the risk of data breaches or AI-based cyberattacks.

Impact on risk management

For companies investing in cyber insurance or cyber risk policies, this formula could become a crucial tool for risk assessment. Insurance companies could use this methodology to more accurately determine cyber insurance premiums, based on objective metrics rather than generic estimates. Additionally, the ability to predict AI behavior could influence coverage conditions, making policies more adaptable to the specific needs of businesses.

Challenges for regulatory compliance

With the implementation of the NIS2 Directive and the DORA Regulation, European companies must ensure robust digital operational resilience. The formula developed by the researchers could facilitate NIS2 compliance and adherence to other regulations, providing a tool to monitor and manage risks associated with the use of offline AI. This is particularly relevant for critical sectors such as energy infrastructure and financial services, where security is an absolute priority.

Applications in the financial sector

In the context of wealth management, the formula could improve crypto wealth management and institutional crypto custody services. Platforms offering institutional custody services could use this methodology to ensure that AI models employed for financial data analysis or asset management do not produce problematic outputs. This is particularly important for crypto AML compliance and adherence to KYC exchange regulations, where precision and security are fundamental.

Considerations for crypto taxation

For professionals dealing with crypto taxation in Italy and bitcoin declarations, the ability to predict AI behavior could simplify crypto tax management. Crypto accountants and crypto tax consultants could use this formula to ensure that AI models used for accounting and income tax declarations do not generate errors that could lead to penalties or compliance issues.

Future prospects

As AI models become smaller and more powerful, the ability to predict their behavior will become increasingly crucial. The formula developed by Johnson and Huo represents a significant step toward creating safer and more reliable AI models, especially in offline environments. With the further development of more powerful hardware and advanced AI models, this methodology could become a standard in the industry, improving security and risk management for businesses worldwide.

Frequently Asked Questions

How much does it cost to implement a monitoring system based on this formula?

The cost depends on the complexity of the system and the specific business needs. However, the authors propose a low-cost monitor that could be easily integrated into existing devices.

How can this formula improve the security of offline AI models?

The formula allows for predicting when an AI model might generate harmful output, enabling preventive measures to be implemented before the model "breaks down" into problematic behaviors.

What are the current limitations of the formula?

Editorial Note and Disclaimer

The guides and content published on GoYou are the result of independent research and analysis activities, for informational, educational, and in-depth purposes.

GoYou does not constitute a journalistic publication or an editorial product pursuant to Law No. 62/2001 and does not engage in real-time information activities.

The GoYou project does not provide professional, technical, legal, or financial advice and disclaims all liability for the improper use of the information published.

In the Crypto sector, every investment involves risks: readers are invited to always inform themselves independently before making any decisions.