The Battle for On-Device AI: When AI Moves from the “Cloud Brain” to Your Pockets and Cars
Hello everyone, I’m your financial journalist. Today, we’re talking about a significant transformation that’s already underway but may not have fully caught your attention: AI is moving from the cloud to your devices.
In the past, when we thought of AI, we thought of ChatGPT, DeepSeek—services that required an internet connection, with servers thousands of miles away doing the heavy lifting for us. But things are changing. Now, phones, cars, and even a robotic duck are starting to have their own “little brains,” capable of working without the need for the internet. This is what we call On-Device AI.
The key takeaway from this article is that 2026 could be the year when On-Device AI becomes commercially viable. Giants like NVIDIA, Meta, Apple, Huawei, Xiaomi, and ByteDance, as well as startups like Mianbi Intelligence and Jieyue Xingchen, are all making significant investments in this field. Why? Because the cloud is too expensive, too slow, and poses privacy concerns, while On-Device AI is cheaper, faster, and offers better privacy protection. This is not just an upgrade in hardware; it’s also a battle for control over how we interact with technology.
Let me break down this complex topic into five key points in plain language:
---
1. Why On-Device AI?
The main reasons are that the cloud is too costly, too slow, and poses privacy risks:
- High costs: Training a large model like Anthropic’s Mythos can cost billions of dollars. Even using it can be expensive; for example, ByteDance’s DouBao service incurs daily costs of 130 to 240 million yuan just for processing requests. On-Device AI runs on your phone’s chip, with almost zero additional costs.
- Latency: Cloud-based AI relies on the internet, which can be unreliable in suboptimal environments (subways, planes, etc.). On-Device AI runs locally, providing immediate responses even without a connection.
- Privacy: Cloud-based AI means your data is transmitted to the cloud, which raises privacy concerns. On-Device AI keeps your data on your device, enhancing its security.
In simple terms, cloud AI is like a professional consultant—powerful but expensive and slow, while On-Device AI is like a handy, always-available “home doctor” that protects your privacy.
---
2. How is it technically possible?
It’s possible thanks to advancements in model compression and chip performance:
- Model compression: Scientists have developed techniques to shrink massive models from trillions of parameters down to a few billion, maintaining their performance in specific tasks (e.g., speech recognition, simple queries, image understanding).
- Chip performance: Chip power doubles every two years (Moore’s Law), and model efficiency also improves rapidly. Modern phones have enough power to run models that used to require supercomputers.
- Example: NVIDIA’s RTX Spark chip can handle 120 billion parameters locally, while AMD and Intel are also increasing their chip performance.
As a result, AI is no longer limited to the cloud; it’s becoming increasingly integrated into phones, PCs, cars, and robots. By 2025, 36% of new phones will have generative AI capabilities, and by 2027, that number will rise to over 50%.
---
3. Who’s involved in this battle?
The battle involves various players with different strategies:
- International giants: They’re buying ecosystems and developing hardware to lower the barriers to AI use (e.g., NVIDIA’s acquisition of Hugging Face).
- Chinese companies: They’re focusing on different aspects of On-Device AI (e.g., Mianbi Intelligence, which sells models to hardware manufacturers).
- Hardware manufacturers: They’re working on integrating AI into their products (e.g., Apple, Huawei).
- Allies and competitors: There’s a mix of cooperation and competition among them, with each trying to control different aspects of the AI ecosystem.
4. The impact of the 399-dollar “robotic duck”
The article highlights NVIDIA’s Microduck, a 399-dollar robotic duck that uses a 2020-era chip and is open-source. It’s a significant milestone because it demonstrates the potential of affordable On-Device AI:
- Low cost: It uses a low-cost chip and is open-source, allowing users to train and customize it.
- Data feedback: Users collect data through the duck, which helps improve the model’s performance.
- Market success: The duck sold out in 4 seconds, showing demand for such affordable AI solutions.
This example shows how On-Device AI is expanding beyond phones and cars to robots, smart homes, and wearable devices.
---
5. The future of AI
The future will see a clear division of roles:
- Cloud models: Responsible for complex tasks and advanced applications.
- On-Device models: For quick, real-time, and privacy-sensitive tasks.
This collaboration between cloud and on-device AI will make AI more accessible and integrated into our daily lives.
For consumers, this means smarter devices that understand us better and can perform tasks more efficiently. For investors, it’s worth focusing on companies developing On-Device chips and models. However, pure third-party model providers may face challenges as manufacturers may start developing their own solutions.
---
In conclusion, the battle for On-Device AI is about democratizing AI, making it more accessible to everyone. While its full potential is still being realized, the combination of advanced chips and efficient models is paving the way for a world where AI is everywhere, in every aspect of our lives.