Summary of Key Points
This article is a practical reflection by an AI product manager. She started with a simple approach of manually creating an AI evaluation workflow for customer service conversations (evaluating each aspect of the model one by one). The company’s evaluation system evolved through three stages: from individual efforts to a more systematic and organized approach. She eventually realized that AI evaluation is not just about collecting data and running scores; it represents a comprehensive strategy that encompasses what to measure, how to measure it, who will perform the measurement, how to use the results, and how to maintain the system in the long term. The article breaks down the core logic of designing an evaluation plan through five critical questions and highlights three common misconceptions, aiming to help readers shift from a focus on execution to a strategic perspective on AI evaluation.
Three Stages of the AI Evaluation System’s Evolution: From Chaos to Organization
The author’s company’s evaluation system went through three phases, corresponding to her personal understanding of the process:
1. Individual Efforts Phase: Each business unit relied on third-party annotators, leading to inconsistent standards (for example, “correct answer” meant different things in different teams). This resulted in low efficiency and slow feedback; the author’s role was limited to producing reports.
2. System Awareness Phase: The company began developing its own system, establishing a complete process from dataset management to annotation, strategy maintenance, analysis reporting, and version control. The author’s role evolved from an executor to someone who could provide optimization suggestions.
3. Automation Phase: Two main issues were addressed: (1) the slow pace of manual annotation due to the need for re-annotating data with each model iteration; (2) variability in scores given by different evaluators for the same conversation. The solution was to implement an automated scoring rubric that clearly defined criteria, allowing tools to perform the evaluation while human reviewers focused on analyzing problematic cases and calibrating the standards.
Five Critical Questions: Building a Navigation Map for Evaluation Strategies
The author divides the design of an evaluation plan into five key questions, each corresponding to a strategic aspect:
1. What to Measure? —— Determine Priorities Clearly
Many people try to evaluate all aspects of the model (accuracy, fluency, security, etc.), but often fail to do so thoroughly. The correct approach is to use a three-tier filtering system:
- Product Development Phases: Verify the model’s functionality during development, assess user experience during deployment, and measure business value in the growth phase (if early stages are not met, there’s no need to test those aspects later).
- Core Use Cases: For example, evaluate whether a customer service bot accurately understands user intentions or whether a code assistant uses correct grammar.
- User Pain Points: Focus on the most critical issues—for instance, a rude response is more detrimental than a slightly slow one.
This leads to the creation of a weighted priority matrix that assigns different weights to each aspect (e.g., critical aspects may account for 60% of the total score).
2. What Dimensions to Use? —— Don’t Rely Only on Correctness
Many teams only evaluate whether an answer is correct, but the AI experience is multi-dimensional. The author recommends using a framework that includes:
- Accuracy (whether the information is accurate),
- Fluency (whether the language is smooth and easy to understand),
- Security (absence of inappropriate content),
- Consistency (no contradictions between responses).
Note that different dimensions require different evaluation methods—accuracy can be automated, while fluency must be reviewed manually.
3. Who Will Perform the Evaluation? —— Assign Tasks Appropriately
Common misconceptions include:
- Asking domain experts to evaluate fluency (experts are valuable, but ordinary users can also provide useful feedback on fluency);
- Having the same group of people evaluate multiple dimensions (which may lead to biased evaluations).
The correct division of labor is:
- Ordinary users: Evaluate fluency and friendliness (which reflect real user experiences);
- Domain experts: Assess professionalism and accuracy (which require specialized knowledge);
- Algorithm teams: Perform automated evaluations on routine dimensions.
4. How to Use the Results? —— Don’t Just Focus on Scores
Scores alone are not useful; instead, use them to:
- Determine if the model meets requirements before moving on to the next phase;
- Identify issues (e.g., low accuracy may be due to insufficient training data or poor model understanding);
- Provide specific recommendations (e.g., suggest improving accuracy through A/B testing or adding more relevant training data).
5. How to Maintain Long-Term Evaluation? —— Don’t Conduct One-Time Assessments
Evaluation should not be a one-off; it should be a continuous process:
- Quick Verification (1–2 weeks): Combine manual and automated methods to quickly identify issues.
- Regression Testing (1–2 months): Store problematic cases from previous iterations and automatically re-evaluate the model after each update to prevent recurrence.
- Regular Operations (long-term): Establish an automated process that runs evaluations before each deployment and regularly calibrates the evaluation criteria.
Three Common Misconceptions to Avoid
1. Overrelying on Automated Scoring: For example, a high score in a knowledge-based test (MMLU) may not reflect a good user experience if the model’s tone is harsh. Solution: Use automation for preliminary screening and manual review for final validation.
2. Misallocation of Resources: Using experts to evaluate aspects that non-experts can handle better, or having the same group of people evaluate multiple dimensions.
3. One-Time Evaluations: Stop evaluating after deployment; ongoing assessments are necessary to ensure model quality over time.
Cognitive Upgrade: From Execution to Strategic Thinking
The author initially thought the issues were related to the complexity of the workflow or the strength of the model, but realized that the real problem was oversimplifying the evaluation process. A proper evaluation system addresses broader questions: what to measure, how to measure it, who will perform the measurement, how to use the results, and how to maintain it in the long term. The key insight is that AI evaluation is not just a tool; it’s a strategic framework, and the evaluation plan itself is an important deliverable.
Next time you’re assigned an evaluation task, don’t rush to create a workflow. Instead, ask yourself the five critical questions outlined in this article. By doing so, you can move from simply executing tasks to developing effective evaluation strategies.