Summary of Key Points
The core argument of this article is that the risks and values of AI do not depend on how intelligent it is (such as the scale of its parameters or the accuracy of its predictions), but on its “position” within the execution chain. The closer it is to actual operations and the more decision-making power it has, the more severe the consequences of any errors will be. Additionally, this position also determines the business model of the AI product; the more critical its role, the less replaceable it becomes. The article uses the analogy of a fish changing roles in different scenarios to illustrate that we need to shift from simply pursuing increased intelligence to carefully designing the boundaries of where AI is used.
Detailed Analysis
1. The Same AI, Different Risks Depending on Its Position
You may have heard claims about AI having an accuracy rate of 99%, but this figure alone is meaningless unless you consider the context in which it is used:
- In a chatbot: A single mistake might just make users think the AI is unreliable, but they can easily overlook it.
- In an internal company knowledge base: A mistake could lead employees to make decisions based on incorrect information.
- Integrated into a customer service system: A mistake could offend customers and damage the brand’s reputation.
- Granted permission to modify production databases: A single error could cause a production line to stop, resulting in millions of dollars in losses.
- Operating industrial equipment: A mistake could potentially lead to a safety accident.
The AI itself remains the same, but the closer it is to real-world operations, the higher the cost of any mistakes. Many people focus on reducing the error rate to 0.01%, but companies should first ask: “What can this AI do right now? Does it just provide suggestions, or can it directly alter data, transfer funds, or operate machinery?” These questions are more indicative of potential risks than accuracy.
2. Capability Determines the Upper Limit, Position Determines the Consequences
We tend to evaluate technology based on its capabilities: a faster computer means better performance, and a smarter AI implies higher intelligence. However, AI is different from traditional software. Traditional software only provides suggestions (e.g., Excel for data analysis), while humans ultimately make the decisions. AI, on the other hand, can take direct action (e.g., sending emails automatically, submitting code, or making payments).
The crucial distinction here is that “capability” refers to what AI can do, while “position” refers to what will happen if it makes a mistake. An extremely intelligent AI with no decision-making power poses limited risks, whereas an ordinary AI with the ability to transfer company funds can be much more dangerous. This is similar to employees in a company: an ordinary employee’s mistake affects a small area, but a CEO’s mistake impacts the entire company—not because the CEO is less intelligent, but because their position is closer to the outcomes that matter.
3. No Matter How Reliable AI Is, Power Should Not Be Granted Arbitrarily
Many people believe that the more reliable AI is, the more power it should be given—e.g., reducing manual reviews when the model performs stably or allowing it to execute tasks automatically. However, this is a misconception: reliability does not equate to controllability.
For example, pilots are highly trained, but aviation still requires checklists; bank employees are trusted, yet banks still have double-checking procedures and transaction limits. The reason is that even the best systems can make mistakes, and when the consequences of such mistakes are significant, boundaries must be established. The same applies to AI: while its intelligence improves gradually (from 99% to 99.9%), the risks associated with power increase dramatically (from a minor chatbot mistake to a major financial error). Just because AI becomes more intelligent does not mean it should be given unlimited authority; boundaries are always more important than capabilities.
4. Position Determines the Business Model of AI
Many AI products have similar functions, such as writing copy, analyzing data, or completing tasks automatically. But why do some remain tools (like chatbots on phones), while others become essential infrastructure (e.g., AI integrated into core business processes)?
The key lies in their position:
- If an AI remains in a dialog box that users can close at any time, it is just a tool and can be replaced.
- If it is embedded in daily workflows (e.g., automatically processing reimbursements or generating reports), it becomes an integral part of the system that the company cannot do without.
- If it sits at a critical point between multiple systems (e.g., connecting CRM and ERP), it has the potential to become a platform.
- Only if all critical operations must go through it can it truly become essential infrastructure.
In other words, functionality determines whether people will use the AI, but its position determines whether they cannot do without it. Valuation should not focus on how intelligent an AI is, but on whether its role in the industry is irreplaceable.
5. A New Standard for Evaluating AI: From “IQ” to “Radius of Power”
In the past, we evaluated AI based on its intelligence—whose reasoning was strongest, whose code was best, or whose multi-modal capabilities were most advanced. But as AI integrates into real systems, companies are more concerned about its “radius of power”—that is, what it can access, change, and the extent to which its mistakes can affect things. Questions like:
- Can it view confidential company data?
- Can it directly modify customer information?
- How many people/ departments will be affected by its errors?
- Are there independent systems that can prevent its misactions?
For example, an AI may be very intelligent but limited in its capabilities; if it can only view data without making changes, its radius of power is small. If it can operate factory robots directly, its radius of power is much larger. Mature systems ensure that AI knows where to stop.
Conclusion
AI is like that fish: it remains the same, but its meaning changes dramatically depending on its position. In the future, we should not only ask how intelligent AI is, but also where we place it. Intelligence determines what it can do, its position determines its power, and power determines the consequences of its actions. What really needs to be designed in the AI era is not just the intelligence itself, but also the “safety distance” between that intelligence and real-world operations.