Summary of Key Points
In the past, when AI was merely an auxiliary tool, the approach of "humans managing AI" (such as manual approval and final confirmation) was effective. However, with the emergence of AI Agents—intelligent entities that can operate continuously, process vast amounts of information, and coordinate tasks automatically—the speed and scale of AI have surpassed human capabilities. As a result, manual confirmation has either lost its value in automation or become a mere formality. Therefore, governance in the AI era requires an upgrade: it's no longer about "humans controlling AI," but rather about establishing a governance structure that involves both humans and AI, with each acting to restrain the other. The core principles include the decentralization of power (to prevent any one entity from holding all authority), the division of responsibilities (with AI handling what it does best and humans taking on irreplaceable tasks), and the clear delineation of responsibilities. It's also necessary to determine who is eligible to participate in governance.
Detailed Analysis
1. Why Does "Human Management of AI" Fail in the Agent Era?
AI Agents are different from previous AI tools. Previous tools required human intervention (e.g., you had to give commands for AI to generate content), but Agents can operate autonomously. For example, an e-commerce Agent can analyze tens of thousands of user data within minutes, interact with multiple systems, adjust prices for hundreds of products, and collaborate with other Agents. If we insist on manual confirmation at every step, two outcomes are likely:
- Automation becomes useless: Humans become a bottleneck, nullifying the efficiency advantages of AI (e.g., an Agent can process 100 tasks in a second, while a human can only confirm 10 tasks in an hour).
- Confirmation becomes a formality: Humans cannot understand all the details performed by the Agent and may simply approve based on a summary, leading to a loss of actual control.
2. AI Is Not Only an Object to Be Managed; It Can Also Help Manage
We often view AI as a source of risk (e.g., potential errors or biases), so our focus is on how to restrict it. However, AI can also serve as a supervisory mechanism:
- One AI can audit the output of another AI (e.g., using Model A to review contracts generated by Model B for errors).
- AI can quickly identify anomalies in hundreds of thousands of transactions (e.g., if an employee suddenly changes the permissions of 100 customers, which may go unnoticed to humans but is immediately detected by AI).
- AI can continuously verify rules (e.g., whether someone bypasses approval processes or alters parameters).
It's important to note that humans are also prone to errors due to fatigue, misjudgment, and influence by interests. Therefore, governance should involve both "humans supervising AI" and "AI supervising humans," as well as mutual oversight between AI systems.
3. Collaborative Governance Is Not About Equal Power for Humans and AI; It's About Leveraging Each's Strengths
Many misunderstand collaborative governance as giving equal voting rights to humans and AI. The key is the division of responsibilities:
- AI's strengths: Processing large amounts of data, detecting anomalies, and ensuring consistency in rules (e.g., continuously checking every transaction).
- Humans' strengths: Setting goals (e.g., defining what constitutes unacceptable risk), handling exceptions (e.g., determining whether employee behavior is a violation or a legitimate adjustment), and assuming legal and ethical responsibilities.
For example, for a large transaction, AI first checks for anomalies, the system verifies compliance with company rules, and then humans decide whether to approve it. None of these steps can be skipped.
4. The Real Danger Is Not AI Having Too Much Power; It's When One Entity Holds Complete Control
Some argue that ultimate decision-making power must lie with humans. However, why should one person have all the authority? For instance, a company administrator could change all rules, reset passwords, and override all AI risk controls and manual approvals. This is just as dangerous as if AI had complete control.
A mature governance system prevents these extremes: it neither allows AI to dominate nor does it give humans absolute power. For example, changing company rules should involve multiple checks (e.g., AI verifying consistency and a team vote), and AI actions must be approved by humans based on value criteria, with another AI confirming compliance. In short, no single entity should have the final say.
5. Collaborative Governance Also Involves Managing Who Can Participate
If a company claims to use AI for governance but the administrator decides who can participate and modify the rules, it's not true collaborative governance, as the authority remains with that one person.
True collaborative governance includes incorporating the governance process itself into the decision-making:
- Adding new AI participants requires a formal process (e.g., team review).
- Changes to AI rules or data scope must be approved by multiple parties.
- Even the definition of what constitutes collaborative governance should be agreed upon collectively.
Otherwise, one person will always be in control of all participants, undermining the principle of collaboration.
6. Responsibility Cannot Be Diluted
While collaborative governance decentralizes power, it does not eliminate responsibility. In the event of an accident, it cannot be blamed on "AI recommendations" or "system automation." Humans must take responsibility for setting goals, approving policies, and managing outcomes. Since AI lacks legal personhood, ultimate responsibility always lies with humans.
In summary, governance in the AI era is about establishing a structure where both humans and AI restrain each other, with clear responsibilities and no one entity holding absolute power. This ensures true security.