Summary of Key Points
After over a year of practical experience with AI coding, Taobao's flash purchase team has addressed the issue of AI-generated code that, although fast, often suffers from poor quality and disorganized structure in production environments. Their approach focuses on making the implicit guidelines in engineers' minds explicit so that AI can follow them. By categorizing requirements, enforcing engineering structure constraints, implementing a quality control loop, and managing the team effectively, they have developed a replicable AI coding solution that not only significantly improves coding efficiency (raising the AI coding rate from 9% to 89%) but also ensures code quality. This transformation redefines the role of programmers in the AI era, shifting them from mere coders to architects and quality gatekeepers.
Three Major Reasons Why AI Coding Fails in Production
Why is AI-generated code often subpar? It's not that AI is incapable; rather, it's because we haven't utilized its potential effectively:
1. Inherent limitations of AI: AI makes predictions based on probabilities, while business requirements are absolute. This is like asking someone who relies on intuition to score a perfect test, which can lead to errors due to misunderstandings. Additionally, AI lacks understanding of the specifics of the business (such as order processing rules) and team coding conventions, resulting in code with logical flaws or misplaced instructions.
2. Lack of effective human-AI collaboration: In the past, we used precise programming languages for coding; now, communicating with AI using natural language can lead to misunderstandings due to different nuances in wording. Moreover, AI can generate thousands of lines of code at once, making it impractical for humans to review each line manually.
3. Cognitive inertia among users: Some are hesitant to use AI for fear of errors or job loss, while others are accustomed to writing specific code and don't know how to guide AI effectively—this is like using an excavator but still digging by hand.
The Key to Overcoming These Challenges: Turning Implicit Guidelines into Explicit Rules Understandable by AI
The solution lies in documenting the implicit rules (such as where to store code and how to handle exceptions) so that AI can strictly adhere to them. They took the following steps:
1. Categorized requirement handling: New and modified requirements are processed separately. New requirements focus on architecture (module boundaries, data flow), while modified requirements emphasize the impact of changes (what needs to be changed and how it affects other parts of the system). This provides AI with appropriate templates to prevent misinterpretation.
2. Establishing an engineering structure framework: They had AI create the project's overall architecture, directories, and technology stack as the highest-level guidelines. For example, by specifying that data-related code must be placed in a "dao" folder, AI learned to organize it properly, which also helped to clarify the messy structure of existing projects.
3. Breaking large tasks into manageable chunks: Since AI has difficulty remembering previous context, they divided large tasks into smaller ones with clear status updates. For instance, when requesting the creation of a file named "66," AI could split the task into smaller sub-tasks and pick up where it left off—once the user entered "continue," AI completed the remaining 5000 lines of code.
Quality Assurance: A Loop of AI Self-Review and Human Final Review
How do we ensure the quality of AI-generated code? Instead of manual line-by-line reviews, they implemented a cycle of AI self-review followed by human validation:
1. AI self-review: They provided AI with a set of review criteria (such as whether business logic is correct and if the code design is sound) to check the code itself. The initial review might only score 70%, but after improvements, it could reach 96%.
2. Human final review: Humans focus on high-risk areas (e.g., potential system crashes or security vulnerabilities) rather than reviewing the entire code, thus improving efficiency while concentrating on critical issues.
3. Iterative refinement of guidelines: If AI makes repeated mistakes (e.g., mixing error handling codes with messages), they update the guidelines directly without modifying the code itself. The more these guidelines are used, the more accurate they become, and AI will learn from them.
Team Adoption: From Pilots to a Self-Sustaining Cycle
Promoting AI coding doesn't involve forcing KPIs; it relies on creating a positive environment and setting examples:
1. Fostering a supportive culture: They regularly encourage successful users to share their experiences and recognize outstanding coders monthly, demonstrating the benefits of AI.
2. Pilot projects: They start with small-scale trials involving 1-2 requirements and individuals, teaching them the entire process before having them promote AI coding within the team.
3. Monitoring metrics: They track two key indicators: the AI coding rate and code quality (e.g., the number of bugs per thousand lines of code and the frequency of code rollbacks) to ensure that efficiency increases without compromising quality.
The Future of Programming
The future of AI coding lies in creating end-to-end closed loops where AI participates throughout the entire process from requirement definition to testing and deployment. This involves two cycles: coding → self-review → optimization, followed by testing → feedback → re-coding. As a result, programmers will evolve into:
1. Architects: They will no longer write code but design systems and establish guidelines (where "guidelines become the new code").
2. Quality gatekeepers: They will focus on ensuring that AI outputs meet business requirements.
3. AI collaborators: They will learn to guide AI effectively, rather than competing with it in terms of coding speed. While AI is faster, human wisdom is needed to direct its efforts.
In summary, the essence of AI coding is to communicate software engineering principles to AI, enabling it to act as an efficient executor while programmers evolve into more valuable decision-makers. This represents not a technological revolution but a transformation in how work is done.