Summary of Key Points
In the field of AI, benchmarks were originally tools used by engineers to evaluate the performance of models during development. However, they have now become a “report card” for model releases, allowing manufacturers to compare their models with those of their competitors and demonstrate their capabilities. As model capabilities become increasingly similar, the credibility and practicality of these scores have sparked a series of controversies: scores are updated monthly without a corresponding improvement in user experience, targeted optimizations may inflate scores, and the testing scenarios often differ from real-world use cases. The industry is adapting to these changes. Some manufacturers have stopped reporting certain scores, others only compare their models with their own past performance, and some have suggested incorporating additional dimensions such as cost (in terms of resources and speed) into the benchmarks. In the future, benchmarks will need to evolve, and people will place more emphasis on the actual capabilities of models rather than just their scores.
Detailed Analysis
1. Benchmarks: From Development Tools to Public Displays
Benchmarks were initially internal tools for engineers, similar to mock tests created by teachers to assess model performance on specific tasks (such as solving math problems or writing code), helping teams identify issues and optimize models. Nowadays, manufacturers use these scores to compare their models with those of their competitors, making it a standard practice in the industry.
Why is this? Because scores provide tangible evidence that can both reassure users about the model’s strength and serve as a means to gain recognition within the industry (like a “ticket to enter the discussion”). For example, when buying a phone, consumers look at performance metrics; similarly, AI manufacturers use scores to demonstrate the superiority of their models.
2. Score Controversies: Why Do People Doubt These “Report Cards”?
As model capabilities converge, the issues with benchmark scores become more apparent:
- Improving Scores Without Improving Experience: Scores are updated monthly, but users often don’t notice a significant difference in model performance. For instance, OpenAI’s SWE-bench score for coding ability only increased from 74.9% to 80.9% over six months, a slow improvement that no longer reflects the capabilities of cutting-edge models, leading OpenAI to stop reporting this score.
- Inflated Scores: Some manufacturers may optimize their models specifically for the benchmark tests (similar to students studying for exams), resulting in high scores that do not reflect the model’s actual capabilities. A Stanford report also noted that independent test results often differ from the scores reported by manufacturers, indicating that benchmarks may no longer be effective in assessing model performance.
- Disconnection from Real-World Needs: Benchmarks often focus on simplistic tasks (e.g., solving math problems), but real-world user needs are much more complex (e.g., writing reports or handling multimodal content). Users question the relevance of high scores when they have no practical benefit.
3. Industry Adaptations: Moving Beyond Score Comparisons
In response to these controversies, the industry is changing its approach:
- Stopping Reporting Outdated Scores: OpenAI has stopped reporting the SWE-bench score because it can no longer accurately assess the capabilities of modern models.
- Focusing on Internal Comparisons: When releasing the Sonnet 5 model, Anthropic compared it only with its own previous versions, avoiding meaningless score comparisons with competitors.
- Incorporating Cost Dimensions: OpenAI scientist Noam Brown suggested that benchmarks should include cost factors, such as the amount of resources required (tokens) and the speed of model execution. Just as when buying a product, consumers consider both quality and price, so should benchmarks take into account these practical considerations.
4. Future Trends for Benchmarks
Controversies will continue, but benchmarks will gradually evolve:
- Upgrading Testing Standards: Future benchmarks will better reflect real-world scenarios, including more complex tasks and cost-related factors. At this year’s ICML (International Conference on Machine Learning) conference, nearly a quarter of the papers discussed the credibility and practical value of benchmarks, indicating that the industry is addressing these issues.
- Scores as a Secondary Criterion: Ultimately, people will evaluate models based on their actual performance, similar to how they evaluate phones—whether they can solve specific problems, are user-friendly, and whether the costs are reasonable.
- The Importance of Development: Although public score controversies exist, benchmarks remain useful for internal development and customer deployments, helping engineers quickly identify model issues, though they will no longer be the sole criterion for public comparisons.
In Conclusion
The controversy surrounding benchmarks reflects the ongoing balance between “metrics” and “actual capabilities” as technology matures. In the future, scores will play a secondary role, and the real winners will be models that can effectively solve users’ problems.