Summary of Key Points
Artificial Intelligence (AI) has evolved from being a tool that merely provides suggestions, such as writing emails or summarizing documents, to an agent that can directly manipulate corporate systems—changing CRM data, initiating refunds, and adjusting cloud configurations. However, existing security systems in companies only address the question of "who can access the systems" and not the new issues of "what actions AI can perform and how those actions should be constrained." Without establishing clear "execution boundaries" and specialized governance models, companies are reluctant to let AI handle critical operations. After all, behind each button lie valuable assets, customer data, and production systems, and mistakes cannot be easily undone.
Detailed Analysis
1. From "Talking" to "Doing": Traditional Security Measures Are No Longer Enough
In the past, corporate security focused on controlling "people"—employee accounts were assigned roles with corresponding permissions; for example, finance personnel could view reports, and IT staff could restart services. This approach worked because all critical actions ultimately involved a human clicking a button, and the system assumed that the person's action was intentional.
With the advent of AI agents, employees grant permissions to the AI, which then performs tasks on their behalf—automatically responding to emails, updating customer status after analyzing tickets, or even executing multiple system-related operations in sequence. The problem is that just because an employee has the permission does not mean the AI should have all the same permissions; just because a human can click a button does not mean the AI can automatically do so as well. Traditional security measures protect against unauthorized access, but now companies need to ensure that authorized actions do not lead to unintended consequences. These are two completely different concepts, and many companies have yet to establish the latter type of security framework.
2. The Most Dangerous Thing Is Not AI Making a Mistake in Speech, But in Action
Previously, concerns were about AI generating false information (hallucinations), but in a corporate context, an AI making a mistake in action is ten times more detrimental than making a mistake in speech. A mistaken statement can be corrected, but an incorrect system modification is irreversible. For instance, AI might apply commands from a testing environment to the production environment or mistakenly execute a refund process instead of just preparing one.
Worse still, AI errors can be amplified along the chain: natural language instructions → AI generates tasks → calls tools → sends API requests → changes business status. The longer this chain, the more risky it is to rely solely on initial authorization. What companies need to control is not whether AI can access the systems but whether it can actually make meaningful changes.
3. Why Does Human Confirmation Not Prevent Risks?
Many companies think that having employees confirm critical actions provides sufficient protection, but this is merely a psychological comfort. For example, the confirmation page displayed to the employee might ask, "Do you want to process these refunds?" However, the actual details of the refund could include refunding the wrong customers, exceeding the amount limit, or bypassing approval processes. Humans see only a summary, while AI executes the complex underlying actions—without knowing what they are actually agreeing to.
This does not mean that confirmation is useless; it simply means that additional measures are needed before execution (e.g., requiring double approval for refunds over a certain amount) and during execution (e.g., ensuring no critical steps are skipped), as well as keeping evidence after the action is completed. Otherwise, confirmation becomes a mere formality.
4. AI Amplifies Vulnerabilities in Old Systems
Humans can also make mistakes, but typically only a few times a day and within a single system. AI, on the other hand, can call dozens of interfaces in minutes, operating across multiple systems (CRM, finance, cloud platforms) without hesitation. If employee permissions are set too broadly, an AI might delete customer records en masse. Although AI is not malicious, its speed, permissions, and lack of awareness of boundaries can turn minor vulnerabilities in old systems (e.g., perfunctory approval processes or unmonitored logs) into major disasters.
5. Companies Need to Establish "Execution Boundaries" to Ensure AI Can Work Without Causing Problems
What companies urgently need is not smarter AI but a set of "execution boundaries"—similar to installing brakes, speed limits, and safety mechanisms for AI:
- Brakes: High-risk actions (e.g., transferring funds) must be stoppable.
- Speed Limits: AI should not be allowed to make unlimited API calls (e.g., sending a maximum of 100 emails per day).
- Trailability: All executed actions should be logged for troubleshooting purposes.
- Guardrails: Critical steps (e.g., refund approvals) must not be bypassed.
- Isolation Mechanisms: Permissions from one system should not be freely applied to another (e.g., CRM permissions should not directly access financial systems).
These boundaries cannot be tied to the business systems, which often prioritize speed. Instead, they need to be independent and implemented before execution (e.g., checking if the amount exceeds limits before API calls or confirming the environment is correct before performing production operations). Only with such measures can companies trust AI to handle critical tasks.
In One Sentence
If AI is to perform actions on behalf of employees, companies must first establish clear rules for what it can do and how it can do so. They need to establish independent execution boundaries and governance models to manage the risks associated with these actions. Otherwise, no one can afford the consequences of uncontrolled AI.
(End of article)