Summary of Key Points
This article uses the 2026 AtCoder programming competition in Tokyo as a case study, where OpenAI’s AI agent defeated human contestants but experienced a three-hour breakdown in the middle. It highlights a critical risk in the practical application of AI: just because an AI has strong capabilities does not mean that every output is reliable. Trial and error are part of its normal process, but real-world scenarios (such as manufacturing and finance) do not allow for such mistakes. Companies cannot rely solely on manual approval or trust in AI’s abilities to avoid risks; instead, they need to establish a “stupid but resolute” set of boundary checks, similar to the competition’s refereeing system, to prevent AI’s errors from causing actual damage.
1. An AI’s “Victory” Does Not Mean Every Step Was Correct
Many people assume that when AI defeats humans, it is omnipotent. However, the details of the competition reveal the truth: OpenAI’s AI struggled with questions D and E for the first two hours, failing repeatedly before finally solving them after three hours. This is not an occasional mistake; it reflects how AI works—instead of providing the correct answer from the start, it approaches the optimal solution through a large number of attempts, feedback, and corrections. Just like students solving math problems, they make numerous drafts (mistaken attempts) before reaching the correct answer. In the competition, these mistakes only affect scores, but in reality, they could lead to production halts or financial losses.
2. Zero Cost of Trial and Error in the Competition, High Costs in Reality
The rules of the AtCoder competition minimize the cost of trial and error: code runs in an isolated environment, and timeouts or errors only result in score deductions without impacting the real world; repeated submissions are not penalized, allowing for corrections. However, business scenarios are different:
- Deploying incorrect code in a production environment can cause system crashes;
- A mistaken transfer of 100,000 (when only 10,000 were intended) could lead to additional losses and difficult recovery;
- Incorrect production instructions may result in wasted materials or delayed deliveries;
- Misconfigured permissions could allow hackers to exploit vulnerabilities.
Failure in the competition is numerical; in reality, it is a disaster—a significant gap between how AI functions in competitions and its use in business.
3. The Stronger AI, the More Hidden the Risks
As AI becomes more powerful, people tend to trust it more and grant it more authority—everything from sending emails to operating production equipment, from making purchasing decisions to handling financial transactions. However, this poses a trap:
- Although the probability of AI making mistakes may decrease, the consequences of a single mistake can be severe (for example, a chatbot’s mistake used to be easily corrected, but an AI-operated industrial device failure could lead to a shutdown).
- People often assume that if AI can handle complex tasks, it won’t make mistakes with routine ones, but success in complex tasks does not guarantee reliability in everyday tasks.
It’s like a top student who occasionally makes simple calculation errors; if that student manages company funds, one mistake could result in a million-dollar loss.
4. Manual Approval Cannot Prevent Execution Gaps
Many companies think that adding manual approval to AI makes it secure, but the AI’s execution process is complex: understanding user intentions, breaking tasks down, calling tools, generating parameters, and executing actions—all of which can go awry:
- You may approve a transfer of 10,000 to Supplier A, but AI might set the parameter to 100,000;
- You may approve an update to the test environment, but AI could mistakenly affect the production environment;
- You may approve the shutdown of a faulty device, but AI could shut down the entire production line.
Manual approval only checks the surface; the underlying issues in the execution process are invisible. It’s like you approving your child to buy apples, only for them to bring back a bunch of rotten oranges—not because the child intended to do so, but due to errors in the middle steps.
5. A “Stupid But Resolute” Boundary Check System is Needed
The competition’s refereeing system is much simpler than AI, but it effectively enforces the rules: if the code times out or makes a mistake, it fails. Companies need a similar system that does not need to understand the complexity of AI’s algorithms or keep up with its rapid updates; it only needs to follow a few fixed rules:
- Does the amount being executed exceed the limit?
- Is the environment for execution a test or production one?
- Does the approved content match what is actually being executed?
- Does the current environment meet the requirements for the task?
This system acts as a barrier: AI finds the “optimal solution,” and the barrier ensures that solution meets the rules. No matter how intelligent AI is, if it violates the rules, the barrier stops it—this is the key to controlling risks.
In Conclusion
The power of AI and its potential failures can coexist. Companies should not expect AI to never make mistakes but rather ensure that those mistakes are caught before they cause real damage. Just as the AI’s three-hour breakdown in the Tokyo competition was allowed for trial and error, in reality, we must prevent such errors from turning into actual losses.