Summary of Key Points
Due to limitations in advanced manufacturing processes, the shortage of high-end HBM (High Bandwidth Memory), and barriers posed by the CUDA software ecosystem, domestic AI chips are shifting from competing on individual card performance to focusing on system-level solutions. This involves connecting numerous “imperfect” individual cards through super nodes and clusters of thousands of cards to create massive computing power capable of supporting large model training and intelligent agent inference. However, this path is not easy: it requires overcoming physical limitations in communication and cooling, as well as addressing the cost associated with software ecosystem migration. Moreover, customers are more pragmatic, prioritizing functionality over cost-effectiveness. Additionally, there is a growing demand for on-device computing power (local devices), leading to a new trend of collaboration between edge devices and cloud services.
1. Why have super nodes suddenly become popular?
The reason for the rise in super nodes is a change in demand:
- Growing models: Modern large models have tens of trillions of parameters and can handle context windows containing over a megabyte of data (equivalent to hundreds of thousands of words), which exceeds the memory and computing power of a single chip. Solutions like Moore Threads’ “256 GPUs working as one card” and Huawei’s Ascend 950’s unified memory addressing aim to overcome the limitations of individual cards by leveraging quantity to achieve higher performance.
- Intelligent agents increasing computational demands: Previously, humans used AI (e.g., asking ChatGPT for information), but now AI is using itself (e.g., intelligent assistants). The consumption of tokens has increased exponentially. Baidu Cloud reports that inference tasks account for a significant portion of domestic computing power requirements, and these tasks are highly concurrent and require long context processing, which can only be handled by super nodes.
However, some skeptics argue that some manufacturers use clusters of thousands of cards merely to showcase their capabilities, as 8 or 16 cards are often sufficient in practical applications, and many such solutions have not yet been successfully implemented.
2. The first hurdle to system-level breakthroughs: Physical limitations
Connecting hundreds of cards is not just a matter of stacking them together; physical constraints must be overcome:
- Communication speed limitations: Data transmission between multiple cards needs to be fast and stable. Copper cables lose signal strength beyond 3 meters, necessitating thicker cables, which lead to increased heat generation and weight. Solutions like Huawei’s Lingqu protocol and Moore Threads’ MTLink aim to address this, but physical limits remain a challenge. Optical interconnects (e.g., Xizhi Technology’s solutions) can extend stable connections for dozens of meters, but they are more expensive and not yet widely adopted.
- Heat and power consumption: Super nodes can consume up to 400 kilowatts of power, with individual cabinets exceeding 15 kilowatts. Air cooling is insufficient, making liquid cooling essential. Although traditional data centers need to be upgraded for liquid cooling, this often requires downtime, which determs many companies. Huawei’s air-cooled super nodes represent a compromise that allows for expansion up to 96 cards without significant infrastructure changes.
3. Software ecosystem: The toughest challenge for domestic chips
Customers do not care about the specifications of chips; they want to ensure they can run their models effectively, which means overcoming CUDA ecosystem barriers:
- Model migration is a labor-intensive process: Many customers’ models and code are based on NVIDIA’s CUDA, requiring rewriting operators when switching to domestic chips. Open-source models are relatively easier to migrate, but closed-source ones require additional development time and cost. Yixing Intelligence notes that migrating models is akin to manual labor.
- Functionality is the primary concern: Customers only care whether the chips can be used; they do not distinguish between GPUs and TPUs and focus on three key factors: card type, price, and quick software integration. Kunlun Chip’s compatibility with CUDA is aimed at addressing this need, but most domestic chips cannot achieve full compatibility, making migration a major obstacle.
4. What do customers really want?
Domestic chip buyers are pragmatic large enterprises, not startups seeking peak performance:
- Functionality first: Many chips are still in the demo stage and fail to work reliably in real environments. Customers test them with their own models and only consider price after verifying functionality.
- Total cost of ownership (TCO) is important: They look at the overall cost, including maintenance, cooling, and migration expenses. For example, high power consumption of clusters results in significant electricity costs, and hiring engineers for model migration is another expense.
- Key customers are industry leaders: Operators, financial firms, and energy companies prioritize compliance, local deployment, and supply chain stability over peak single-card performance. In Huawei’s use cases, customers focus on reducing processes by 30 seconds or lowering response times by 80%—these business metrics are more important than benchmark scores.
5. The rise of on-device computing power
On-device computing (e.g., AI PCs and home AI hubs) accounted for 30% of the exhibition at WAIC this year, highlighting a new trend of collaboration between edge devices and cloud:
- Privacy and latency considerations: On-device solutions are necessary for applications like emotional companions robots that require private user conversations to be processed locally.
- System integration: Companies like Cixin Technology integrate CPUs, GPUs, and NPU on their platforms and pre-install intelligent agent operating systems. Moore Threads’ AIBOOK and AICUBE represent examples of this trend, with the latter serving as home AI hubs. This division of labor allows cloud services to handle large model training while on-device devices perform local inference.
The rise of on-device computing indicates that computing power is not solely concentrated in the cloud but is distributed between both cloud and edge devices, providing another pathway for domestic chips.
6. Can domestic chips break through?
The system-level approach is essential for domestic chips to compete with NVIDIA. While individual cards cannot match NVIDIA’s performance, they must leverage collective strength. However, this path is fraught with challenges: physical limitations, software migration, and cost issues. The real test of a chip’s effectiveness comes in the customer’s data center, where it must perform stably. To succeed, domestic chips need to ensure functionality before focusing on reducing costs, rather than immediately competing on peak performance.
In summary, while domestic chips face significant challenges, the system-level approach is the only viable path to breakthrough. By working together and overcoming these obstacles, they can gain a foothold in the market.