Summary of Key Points
Weco’s AIDE2 experiment is the first rigorously verified case of “Level 1 Recursive Self-Improvement (RSI):” It utilizes a structure where an “outer Agent” improves an “inner Agent,” continuously finding more efficient ways for the inner Agent to perform tasks under strict constraints such as fixed costs, multi-task testing, and private scoring methods. While it has achieved improved performance over manual tuning, it has not yet reached the level of “self-ignition” (the ability to improve its own improvement process) or intelligent explosion. The value of this experiment lies in transforming the vague concept of “AI self-improvement” into a quantifiable and reproducible scientific issue, which offers practical guidance for industrial research and development. However, it is still far from the scenario of “AI self-evolution” depicted in science fiction.
How Does the Experiment Work?
The structure of AIDE2 resembles that of a coach training an apprentice:
- Inner Agent: The agent responsible for the actual tasks, such as code modification, experiment execution, and task optimization (e.g., tuning machine learning models or performing combinatorial optimizations).
- Outer Agent: The “coach” that does not directly solve problems but focuses on the inner Agent’s working methods (e.g., search strategies, context organization, debugging rules). After rewriting these rules, it uses the updated inner Agent to conduct tests; if the results are better, the new rules are adopted; otherwise, they are discarded.
Key Detail: AIDE2 does not modify the underlying models (such as the weights of Claude or Gemini) but only adjusts the Agent’s code and logic. Therefore, it is more akin to an “automated research and development engineer” rather than a general intelligent agent that can train itself—the former can help companies save on manpower, while the latter represents the self-evolution depicted in science fiction.
Why Are the Results Reliable?
Many claims of “self-improvement” are actually deceptive (e.g., focusing only on specific types of problems or using excessive resources). AIDE2 addresses these issues with three rules:
1. Public vs. Private Scores: The inner Agent can see its “simulated test scores” to adjust its approach, but whether the changes are adopted depends on the “final scores” (which are private and not visible to the inner Agent), preventing it from making superficial improvements.
2. Fixed Costs: Each evaluation is subject to a budget limit in dollars for model calls and computing resources. The Agent cannot rely on endless trials or additional machines to bypass these constraints; it must use its resources more efficiently.
3. Multi-Task Testing: The inner Agent must handle various tasks, including machine learning, combinatorial optimization, and agent infrastructure development, ensuring that agents specialized in only one type of task are not favored.
These three constraints together ensure that the improvements are genuine and not merely a form of “self-presentation.”
What Levels of RSI Does AIDE2 Achieve?
Weco has categorized RSI into four levels (not industry standards but for clarity):
- Level 0: The agent can automatically complete the research and development process, albeit less efficiently than humans.
- Level 1 (achieved by AIDE2): Continuous improvement (not a one-time occurrence), better performance than humans, and the ability to transfer skills to new tasks without unlimited cost increases.
- Level 2 (“Self-ignition”): The improved inner Agent can also improve the outer Agent’s self-improvement capabilities (e.g., an apprentice becoming a better coach).
- Level 3 (Turning Point): Accelerated progress within a fixed budget, potentially leading to intelligent explosion.
AIDE2 did not pass the Level 2 test: Although the improved Agent performed faster initially, its ultimate performance was similar to that of a human coach, with no sustained improvement in its self-improvement ability.
What Is the Industry Concerned About?
For companies, the concept of intelligent explosion is far-fetched; more immediate issues are practical concerns. The optimized Agent code created by AIDE2 is highly complex and acts like a “black box”—difficult for humans to understand, maintain, and troubleshoot when errors occur. For example, with AIDE85, if there is dead code, the team must address the following questions:
- How to track the origin of each modification?
- How to reproduce the experiments that led to successful results?
- How to isolate problematic modifications?
- How to explain to customers a solution generated by the machine but not understandable to humans?
The core capability for future research and development teams will not be the ability to let agents self-modify, but rather the ability to design rigorous experimental environments (e.g., using private scores and fixed costs) that prevent fake improvements and allow for corrective actions when agents deviate from the intended path.
Significance and Limitations
Significance: For the first time, a scientific method has demonstrated that agents can continuously find better ways to perform tasks under strict constraints. This forces research and development teams to reconsider their approach: in the future, agents will not only be products but also subjects of ongoing improvement.
Limitations:
- The underlying model weights were not modified, so it is still far from true self-evolution.
- The code is poorly understandable, leading to high maintenance costs.
- The results are based on company reports and require independent verification.
While AIDE2 does not represent an “AI revolution,” it points the way for turning vague AI capabilities into practical tools through rigorous experimentation.
In Conclusion
AIDE2 has proven that AI can assist in improving the ways humans perform tasks. However, it has not yet demonstrated that AI can improve its own self-improvement abilities. This is a small step forward, but it brings us one step closer to achieving “controllable AI self-improvement.”