Hello! I'm your financial analysis assistant. Although the title of this article sounds a bit philosophical, it actually hits upon a critical issue in the current digital transformation of businesses, especially in the implementation of AI: we place too much faith in "identity authentication" and overlook "behavior control."
To help you easily understand this lengthy article, I've broken it down into five key sections and explained the underlying business logic in plain language.
---
Core Summary: Stop Asking "Who He Is" and Ask "What He Is Doing"
The main point of this article is quite sharp: traditional business management and IT systems are based on the assumption that "as long as a person or program has a legitimate identity and the correct permissions, everything they do is correct." However, in the age of AI, this assumption has collapsed. Because AI agents possess the speed and scale of machines, they make mistakes much faster than humans can detect them. Therefore, future corporate security cannot rely on "trusting a perfect person or machine"; instead, it must rely on a "structure" that can tolerate errors and limit the extent of damage. In other words, we need to shift our focus from "who has the authority" to "what actions are allowed."
---
In-Depth Explanation: Five Dimensions Explained in Simple Terms
1. Acknowledging That Everyone Makes Mistakes is a Sign of Advanced Institutional Wisdom
[Original Viewpoint] A mature system does not reward perfection but recognizes the limitations of human nature.
[Plain Language Explanation] When we were young, we thought "no one is perfect" was a sign of tolerance, but as we grew up, we realized this is actually a form of risk control. In hospitals, even the most experienced doctors have their prescriptions reviewed by pharmacists, and even the most skilled pilots go through checklists before taking off. Why? Because systems protect not only against bad actors but also against people who are tired, make mistakes, or have incomplete information. If a company expects every employee or administrator to be always sober and rational, it would have failed long ago. Truly excellent companies assume that everyone will make a mistake at some point and design processes to mitigate the consequences.
Conclusion: Imperfection is not scary; what's scary is building a system on the fragile assumption that no one ever makes a mistake.
2. Mistakes Are Not Beautiful, but Structures That Can Withstand Them Are
[Original Viewpoint] Don't praise flaws; instead, praise the beauty of systems that can accommodate them.
[Plain Language Explanation] Don't fall for the notion of "imperfect beauty." A wrong transfer is just a wrong transfer, and a production accident is just an accident—no one finds that romantic. But why are bridges stable? Not because they believe steel never gets tired, but because they have redundant designs. Why are databases reliable? Not because they assume every write operation is successful, but because they have rollback mechanisms.
Key Shift in Thinking:
- Old Thinking: How can we prevent the subject (human/machine) from making mistakes at all? (This is the pursuit of perfection, which is difficult to achieve.)
- New Thinking: If the subject will definitely make mistakes, how can I design my system to make it difficult for them to break through all defenses at once? (This is the pursuit of resilience, which is more realistic.)
Conclusion: True security is not about eliminating uncertainty but about putting it within bounds so it can't escape.
3. Current Systems Have a Major Bug: Legality Does Not Equal Correctness
[Original Viewpoint] Identity confirms "who you are," and permissions determine "if you can do something," but they don't address "whether you should do it at this time."
[Plain Language Explanation] This is the most poignant part of the article. Modern corporate IT systems mainly issue "IDs" and "access cards." For example, a CFO has the authority to make payments (legitimate identity, legitimate permissions). But if they type the wrong account number or get deceived, the payment is still incorrect. The system only recognizes that they are a CFO, not whether the payment to this stranger is justified. In the past, such flaws were addressed through manual processes like meetings, phone calls, signatures, and waiting. These seemingly inefficient procedures were actually security buffers. However, with AI, these checks are eliminated, exposing vulnerabilities.
Conclusion: Being legally authorized does not mean acting correctly. Current systems easily mistake being qualified for doing something with actually doing it correctly.
4. AI Didn't Invent Mistakes, but It Gives Them "Rocket Propulsion"
[Original Viewpoint] AI gives imperfect entities the speed, scale, and continuity of machines.
[Plain Language Explanation] In the past, people made mistakes a few times a day, and they had to stop to think; errors spread slowly, giving managers time to detect and stop them. AI agents are different: they can complete what would take humans hours in seconds, operate on thousands of accounts simultaneously, and make autonomous decisions. This leads to a terrifying situation where the speed of error spread systematically exceeds human observation and correction capabilities. In the past, we could say "someone will be monitoring," but now that's no longer possible. If AI clears a company's accounts in 3 seconds and you only realize it 3 hours later, it's too late.
Conclusion: The greatest risk with AI is not its stupidity but its speed and endurance. It gives errors the power of machines.
5. The Core Competitiveness of Future Businesses: From Trusting People to Trusting Structures
[Original Viewpoint] We need to shift from managing identities to managing actions, states, and boundaries.
[Plain Language Explanation] This is the ultimate advice for CEOs and CTOs. Traditional management logic is based on "because this is a trusted person/system, it's allowed." Future logic should be based on "because this action complies with boundaries, is in the correct state, and has sufficient evidence." This means businesses need to establish new "trustworthy structures":
1. Separate Decision-Making from Execution: Let AI handle execution while another mechanism or person makes the decisions.
2. Limit Actions: Even the most powerful AI can only act within specified boundaries.
3. Verify the Reality: Before each action, the system must confirm whether the current situation supports it (e.g., are there enough stocks, sufficient account balances?).
4. Retain the Right to Overrule: Regardless of who it is, if a red line is crossed, the system must have the ability to stop immediately.
Business Implications:
When all companies can access the same advanced AI models (making AI capabilities as basic resources), the one that dares to let AI work effectively without causing major issues will win. This requires a more radical philosophy of automation: I don't expect AI to be perfect, but I design a system that ensures it won't ruin the company, even if it's not perfect.
---
For Everyone:
1. For Professionals: Don't rely on your personal reliability to secure your job. In the future, your value will lie in your ability to design systems that can mitigate your mistakes.
2. For Entrepreneurs/Managers: Review your company's processes. Relying on "boss supervision" or "experienced employees" for security is very risky in the AI era. Invest in technologies that can monitor behavior, restrict permissions, and automatically verify conditions.
3. For Investors: Look for companies that provide not only AI capabilities but also "AI security measures," "behavioral audits," and "automated risk control." The future's competitive advantage will not come from how smart the models are but from how robust the underlying structures are.
In One Sentence: No one is perfect, and neither are machines. In the future business world, the ones who succeed will be those who build systems that can tolerate errors, not perfect individuals.