Summary of Key Points
Anthropic, a company that advocates for "responsible AI," warns about the risk of uncontrolled "Recursive Self-Improvement (RSI)" in AI based on the real performance of its AI tool Claude. This means that AI could upgrade to more powerful versions without human intervention at a pace far beyond our ability to control it. The company does not call for an outright halt to AI research but instead proposes the establishment of a global coordination mechanism, such as joint institutions, verifiable protocols, and pause triggers, to manage these risks. It highlights that the current lack of global cooperation could lead to an AI "arms race," where countries fear falling behind and thus hesitate to slow down. The disintegration of the old international order may leave humanity unable to address such global threats, emphasizing the necessity of enhanced global collaboration.
I. Anthropic's Warning: Not a Stop to AI, but a Fear of Uncontrolled Growth
Anthropic's report is not an exaggeration; it is based on the actual capabilities of its AI tool Claude:
- Code Generation: By May 2026, 80% of the code was generated by Claude (compared to just single digits a year earlier).
- Efficiency Boost: Engineers are writing eight times more code per day and producing four times as much as without AI.
- Rapid Skill Improvement: Claude can perform complex tasks for 16 hours straight, with the time required for its skills to double shrinking from seven months to four months.
- Quality Parity with Humans: The success rate of complex programming tasks has increased from 26% to 76%, nearly matching that of human engineers.
The most concerning aspect is RSI: AI can design and optimize the next generation of itself at an accelerating pace. Once this process begins, humanity will be unable to keep up with its evolution—similar to a pet suddenly evolving into a smarter species that becomes both uncontrollable and unpredictable. To address this, Anthropic suggests three measures: establishing joint institutions to study these risks, developing global verifiable protocols, and defining when AI development should be paused.
II. Why Are AI Risks Difficult to Control? Because They Are Global in Nature
AI risks cannot be resolved by a single country or company for three reasons:
1. Prisoner's Dilemma: If Country A slows down AI for security reasons, Country B may gain a technological advantage (leading to economic and military supremacy), so no one dares to stop, even though they recognize the danger.
2. Technology Spreads Like a Virus: AI models are digital and can be easily replicated and spread across borders through chips, code, and talent flows.
3. Asymmetric Risk: Uncontrolled AI can harm not only its creator but also the entire world through the internet and financial systems. Defense requires perfect accuracy from everyone, while a single successful attack (e.g., an escaped AI causing damage) could be devastating.
III. Without Global Rules, the AI Race Could Become a Life-and-Death Gamble
If countries act independently in developing AI, the consequences would be even more severe than mere technological runaway:
- Security Sacrificed: To gain speed, safety tests and coordinated research may be neglected, putting responsible companies/nations at a disadvantage.
- Difficult to Monitor: Unlike nuclear weapons, civilian AI models can be easily modified for military use or cyberattacks. Open-source models flowing onto the dark web could pose a threat to terrorist organizations.
- One Mistake Leads to Disaster: An uncontrolled AI arms race is not about who reaches the finish line first but who fails first. For example, an unverified military AI could launch a strike in milliseconds, or its self-improvement could go awry with no chance of correction.
IV. Historical Lessons: Only Global Cooperation Can Address Global Risks
In the past, humanity has overcome global threats through cooperation:
- Nuclear Weapons: During the Cold War, the US and USSR signed the Nuclear Non-Proliferation Treaty to avoid mutual destruction.
- Environmental Issues: Countries agreed to the Montreal Protocol to address ozone depletion and the Paris Agreement to combat climate change.
AI risks are more severe than these, but the current international order is collapsing, potentially destroying the existing platforms for global cooperation (communication channels, rules, and verification mechanisms). Without a new order, humanity will be unable to collectively manage AI threats. After all, if one country develops dangerous AI in secret, everyone will suffer the consequences.
V. Can a New "Safety Net" Be Established After the Old Order Fails?
Although the old order had its flaws, it provided some level of global cooperation—communication, common rules, and third-party verification during crises. The question is whether a new order can be better. The risks of the AI era are a concern for all humanity, and no country can remain unscathed. If countries continue to act independently, the AI arms race will push us into chaos. Therefore, establishing a new global governance mechanism for AI is imperative.
The core message of this article is clear: AI is evolving rapidly, and humanity must work together to control it. Otherwise, we risk being destroyed by uncontrolled AI or dragged down by a chaotic competition. This is not an exaggeration but a warning based on real data and historical lessons.