Summary of Key Points
This article highlights the significant shift in AI from being a “content generation tool” to a “real-world execution system.” In the past, AI errors could be corrected (for example, if a text was incorrectly written, it could be regenerated). However, now, such errors can lead to irreversible consequences (such as transferring the wrong amount of money or deleting critical data). The traditional approach of measuring AI performance using metrics like “99% or 99.9% accuracy” has a fatal flaw: it only counts the number of errors and ignores their consequences and potential extreme risks. The article analyzes the key aspects of AI execution safety from seven mathematical perspectives: errors are not equivalent; a history of no accidents does not mean no risk; rare incidents can be more costly; multiple approvals require independent judgment; AI confidence levels do not equate to actual safety; the final line of defense should be simple and conservative; and efforts should focus on both reducing and limiting errors.
Detailed Analysis
1. Don’t Be Misled by High Accuracy Rates – The Cost of Errors Matters More Than Their Frequency
We often assume that an “99.99% accuracy rate” means safety, but this figure only counts the number of errors, not what those errors entail. For instance, an AI that mislabels an email and one that transfers 100 million yuan in the wrong account are both considered “one error” from an accuracy perspective, yet the consequences are vastly different.
In mathematical terms: **Risk = Probability of Occurrence × Potential Loss.* Even if the probability of an error is very low (e.g., one in ten thousand), a loss of 100 million yuan per occurrence would represent a significant risk that a company cannot afford. It’s crucial to note that a company may be able to handle small, regular losses, but a single catastrophic error could lead to bankruptcy. The mathematical concept of “average values canceling out risks” does not apply in the case of a fatal mistake.
2. “No Accidents Yet” Does Not Equal Safety – Small-Risk Events Can Be Hidden
Some argue that if a system has operated without issues for ten thousand times, it must be safe, but this only indicates that no risk has been triggered so far; it does not prove that the risk doesn’t exist. The probability of no accidents occurring after many tests is calculated as: (1-p)^n (where p is the probability of an error and n is the number of tests). For example, with a probability of 0.01% and ten thousand tests, the likelihood of no accidents is approximately 90%. This means that even if there is a risk, it might not be detected in ten thousand trials.
Many major accidents occur after the system has been functioning “stably” for a long time because the risk was always present but just hadn’t been triggered. The absence of observed risks does not mean they don’t exist.
3. Rare Errors Can Be the Most Dangerous – Extreme Scenarios Are Crucial
Frequent minor errors (such as page lagging) are relatively easy to fix, but more serious issues that occur once every few years can be devastating. For instance:
- Issue A: 1% chance of losing 100 yuan (average risk: 1 yuan)
- Issue B: 0.001% chance of losing 10 million yuan (average risk: 1 million yuan)
Issue B is a hundred times more risky, and the potential loss is much higher due to its rarity. Similarly, in an AI execution system, a minor issue might affect user experience, but a single major error in critical scenarios (such as cross-border transactions or emergency operations) could be catastrophic.
Safety assessments should consider how likely errors are in high-value, unfamiliar contexts, or when multiple rules conflict.
4. Multiple Approvals Are Not a Panacea – Independent Judgment Is Essential
Companies use multiple approvals to reduce risk, but this only works if the reviewers make independent decisions. If three reviewers each have a 10% error rate, the probability of all three making the same mistake is only 2.8%, which is much lower than with a single reviewer. However, if all three rely on the same AI-generated summary without checking the original data, they could repeat the same error. Adding more approval layers does not necessarily improve safety if the reviewers use the same information.
5. The Final Line of Defense Should Be Simpler – Simple Rules Are More Effective
Since other methods have limitations, the final line of defense should be based on simple and conservative rules. While AI handles complex tasks, the security system only needs to check a few basic conditions: does the amount exceed limits? Has the destination address been verified? Are there enough authorized users? Is the device functioning properly? If any of these conditions are not met, the execution should be denied. The system doesn’t need to understand “emergency situations” or be convinced by “convincing reasons”; its role is to prevent errors from becoming irreversible.
In Conclusion
Probability can assist AI in making decisions, but it should not have full control over executions. The essence of safety lies in assuming that errors will happen and using simple rules to prevent them from causing irreparable damage.