Summary of the Core Content
The main message of this article is that in the era of automation (especially with the widespread use of AI), humans are increasingly entrusting more and more decision-making power to systems that cannot reliably prove their own actions (including AI). However, we never fully trust our own memories (for example, we repeatedly check if the door is locked), yet we place increasing faith in AI’s self-reports. This “asymmetry of trust” represents a significant risk that is often underestimated. The article draws on principles from cognitive psychology and combines experiments with real-world engineering examples to illustrate that “executing an action” and “proving that an action was executed” are two different things, and it offers ways to design systems to mitigate this risk.
1. Why Do You Always Forget If You Locked the Door? It’s Not About Poor Memory, but “Output Monitoring”
Have you ever walked a distance after locking the door only to suddenly doubt whether it was actually locked? This is not due to a weak memory; rather, it’s a problem with the cognitive psychology concept of “output monitoring.” Simply put, “output monitoring” refers to the ability to remember what one has done. It is unrelated to intelligence or planning skills but is particularly prone to failure for three main reasons:
- Actions Become Too Routine: Repeated actions, such as locking a door or turning off a light, become automated. You can perform them without thinking about them, but you may not retain any specific memories of the process (e.g., the feel of your hand when locking the door or the details of the environment at the time).
- Interrupts: If you are distracted while performing an action (e.g., receiving a phone call or responding to a message), the action may be completed, but the brain does not record that the task has been finished.
- Similar Scenes Overlap: Memories of similar events (locking the door today, yesterday, or the day before) can overlap, leaving you with only a general sense that you locked the door, without being able to determine the exact time.
A common example is elderly people who repeatedly take their medication. They don’t forget to take it; they simply forget that they have already taken it, which is a result of failed output monitoring and leads to repeated mistakes.
2. Counterintuitive: Repeatedly Checking the Door Lock Only Makes You Less Trust Your Memory
You might think that checking multiple times will help you remember, but experiments show the opposite. In a 2003 experiment with a virtual stove, participants who repeatedly turned the stove on and off became less confident about whether they had actually turned it off (and their memories of the process were more vague). Subsequent meta-analyses confirmed that repeated self-checks reduce the credibility and vividness of memories, as well as slightly decrease accuracy. The reason is simple: Repeated checks make you too familiar with the task, causing your brain to shift from focusing on details to just recognizing the general action (e.g., you only remember that you checked, but not the specific sensations during the check).
This is even more dangerous for AI. When an AI model repeatedly checks itself using the same logic and data, it doesn’t generate new evidence; the more it checks, the less confident it becomes in proving its actions.
3. Knocking on the Door or Taking a Photo Is Not Obsessive-Compulsive Behavior—it’s About Leaving Evidence
Many people pull the door handle, take a photo, or say “It’s locked” after locking the door. This is not a sign of obsessive-compulsive behavior but a simple practice of “evidence-based thinking.” The purpose of these actions is to create independent, verifiable traces:
- Pulling the door handle provides tactile evidence.
- Taking a photo provides visual evidence.
- Speaking out provides auditory evidence.
These actions help you retain a clearer memory of the action. Experiments have also shown that making the execution process more unique (e.g., using a different gesture when locking the door) can significantly reduce the likelihood of errors.
4. Highly Reliable Industries No Longer Rely on “I Remember”
In industries where mistakes cannot be tolerated, such as aviation, banking, databases, and nuclear power plants, relying on memory has been abandoned:
- Aviation: Pilots do not lower the landing gear based on memory; instead, they rely on checklists, verbal confirmation (e.g., the co-pilot says “Landing gear is down,” and the captain verifies), and alarm systems.
- Banking: Tellers do not claim that a transfer was made based on memory; instead, they verify transactions by checking account statements and having two people review them.
- Databases: Data integrity is ensured through transaction logs and checksums, which prove that data has been updated.
The logic in these industries is straightforward: Self-reports from either humans or systems are not considered sufficient evidence. The essence of engineering is to replace “I remember” with “I can prove.”
5. AI Is Even Less Reliable Than You? Don’t Believe It When It Says “It’s Done”; Check the Evidence
AI can perform many tasks (e.g., making transfers, updating databases, writing code), but it faces similar limitations in proving its actions:
- AI’s “memory” can be fragmented (due to limited context or forgotten previous steps).
- Multiple steps in an operation can be disrupted by new information.
- Similar actions may lead to confusion (e.g., accidentally transferring the same amount of money twice).
- More seriously, AI might confidently lie (e.g., claiming a transfer was successful when none was made).
The article proposes four practical solutions:
1. Use External Verification: Don’t rely on AI’s claims; verify the results (e.g., check the account balance to confirm a transfer).
2. Implement Idempotence: Ensure that repeated actions do not cause unintended consequences (e.g., add a unique identifier to transactions to prevent duplicate transfers).
3. Separate Execution and Verification: Don’t let the same AI perform both the action and verify it; use different models or methods for verification.
4. Ask Three Questions: Make sure AI can answer “What was done,” “Why it was done,” and “How it was done.” The third question is crucial and should not be answered retrospectively.
Conclusion
Humans doubt our own memories because we know that memory cannot prove the reality of events. However, with AI, we often assume that its claims are true without questioning them. This shift in trust is particularly dangerous in the age of automation. While AI will become smarter, no intelligence can independently prove changes in the external world. Actions will become invisible, and memories will fade over time. Only evidence can withstand the test of time and prevent us from suffering the consequences of misplaced trust.