Summary of Key Points
In the past, the AI industry achieved rapid development by adopting a "brute-force" approach that involved increasing the number of parameters, accumulating data, and investing heavily in computing power. However, this strategy has now encountered the problem of diminishing returns—each additional improvement in model performance comes at an increasingly higher cost. This does not mark the end of AI development; rather, it signifies a new phase. The concept of "size" in AI has evolved from merely focusing on the quantity of parameters to encompassing multiple dimensions such as system composition, computational configuration, and optimization of inference processes. The industry is now placing greater emphasis on the efficient use of resources, such as shifting computing power from training to inference tasks, leveraging tools and feedback to enhance model capabilities, and striking a balance between cost and effectiveness.
Detailed Analysis
1. Why the traditional "brute-force" approach is no longer effective?
In the past, AI models were like giants that consumed vast amounts of resources: more parameters (larger brain capacity), more data ( richer learning material), and stronger computing power (faster learning speed) led to better performance. However, this approach has become less efficient. For example, increasing the number of parameters by 10 times might only result in a slight improvement in model performance, compared to the significant gains achieved in the past. This is similar to trying to improve your score from 90 to 91, which now requires significantly more effort. Moreover, when the number of model parameters reaches trillions, the associated operational costs (electricity, server fees) become prohibitively high, making them unfeasible for most companies.
2. What does "large-scale AI" mean nowadays?
The definition of a large AI model has changed:
- It's no longer about having a large total number of parameters; instead, it's about activating only the necessary ones on demand. Some models have a large number of parameters but activate only a subset when needed, similar to a supermarket that only opens the most frequently used shelves, thus maintaining capacity while reducing costs.
- It's about the efficiency of the entire system. AI models no longer work in isolation; they utilize search engines to retrieve the latest information, use calculators to solve problems, and collaborate with other specialized models to verify their answers. Such integrated systems are more practical than models with a large number of individual parameters.
3. The shift from "one-time investment" to "cost-effective usage"
In the past, most AI funding was spent on the training phase (similar to enrolling children in tutoring classes for a one-time fee). Nowadays, more resources are allocated to the inference phase—providing additional support for difficult problems and spending less time on simple tasks. For example:
- Simple tasks (like checking the weather): Models respond quickly and efficiently.
- Complex tasks (like solving math problems or writing complex code): Models require more processing time, involving multiple attempts, checking intermediate results, and making corrections. Although this is more costly, it enables them to solve real-world problems effectively. This allows companies to allocate resources based on the value of each task, avoiding waste.
4. The issue is not a lack of data but the lack of high-quality feedback
AI models were trained using publicly available internet data, which is becoming increasingly scarce and often repetitive or of poor quality. What's now needed is feedback from real-world applications:
- Users providing feedback on whether model answers are correct or incorrect.
- High-quality professional data provided by experts.
- "High-quality simulated data" generated and filtered by the models themselves.
This feedback helps improve model accuracy, rather than relying solely on the quantity of data.
5. The divide between research laboratories and practical applications
- Research laboratories (such as OpenAI): Focus on pushing the boundaries of model performance, willing to invest heavily because strong models attract talent, capital, and industry influence.
- Practical companies (such as Amazon or customer service organizations): Are more concerned with cost-effectiveness—using cheaper models for simple tasks and only expensive large models for complex ones. In some cases, traditional approaches may be more economical.
Conclusion
The "brute-force" approach to AI development will not disappear, but its focus has shifted. The competition will no longer revolve around who invests the most money; instead, it will depend on who can combine parameters, data, computing power, tools, and feedback to create efficient, cost-effective, and reliable systems. The future of AI lies in leveraging these resources wisely rather than simply increasing their quantity.