Summary of Key Points
Mainstream large language models (such as ChatGPT) face performance bottlenecks when dealing with long texts—documents consisting of millions of words or code repositories—due to the "quadratic computational complexity" inherent in the Transformer architecture. The startup Subquadratic introduced the SubQ model, which uses "dynamic sparse attention" to reduce the computational load from a square increase with text length to a linear or logarithmic increase. Third-party tests have shown that SubQ can efficiently process long texts containing tens of millions of tokens at a fraction of the cost of traditional models. However, there are controversies surrounding SubQ, such as its reliance on open-source model parameters and its limited availability to the general public, which could potentially impact the future development of large language models.
I. What exactly is the "bottleneck" for large models when processing long texts?
Have you ever tried asking AI to read a 1000-page contract? It either provides irrelevant answers or simply states it can't handle the task. This isn't due to laziness but rather a mathematical limitation in its "memory capacity." The Transformer architecture used by mainstream models requires calculating the relationships between each word (token) and every other word in the text to understand its meaning. For example, if there are n words in the text, the computational cost is n×n (quadratic). Thus, doubling the text length quadruples the computational demand, and increasing it tenfold increases it by a hundredfold.
For instance, processing 1 million words (about 500,000 tokens) would involve 2.5 trillion calculations, which would take several minutes on a GPU at a cost of several dozen dollars. Handling 12 million words (6 million tokens) would result in an enormous computational load of 360 trillion calculations, making it prohibitively expensive. As a result, the "context window" (the maximum length of text that commercial models can process) is typically limited to 120,000–200,000 words, which means they struggle with annual customer service tickets or code repositories containing hundreds of files.
II. How does SubQ reduce unnecessary calculations?
Traditional Transformers use "dense attention," where each word needs to be compared with every other word, even those that are irrelevant. SubQ employs "dynamic sparse attention," which focuses only on calculating meaningful relationships and skips the insignificant ones. Unlike previous approaches that used "fixed sparsity" (e.g., considering only the preceding and following 500 words), SubQ includes an "intelligent filter" that learns which words are relevant to the current word. For example, in the sentence "The animal didn't cross the road because it was scared," the model would automatically determine that only "animal" needs to be considered, eliminating the need to calculate relationships with "road" or "scared." This reduction from n² to n×logn (e.g., from 1 trillion calculations to 16 billion calculations for 1 million words) enables SubQ to handle tens of millions of tokens efficiently.
III. How effective is SubQ in practice?
Third-party tests have yielded the following impressive results:
1. High accuracy in long-text retrieval: SubQ accurately finds key information in texts with 6 million (about 12 million words) or 12 million tokens, with an error rate of just 2%.
2. Fast speed: It is 56 times faster than earlier sparse models.
3. Low cost: While Anthropic's Opus model costs $2,600 to perform the same task, SubQ only costs $8—a 300-fold difference.
4. Competitive performance: SubQ scored 89.7% in programming tests, placing it on par with top models from OpenAI and Google.
In short, SubQ can process long texts that were previously unmanageable, doing so quickly and at a low cost without any loss of functionality.
IV. What are the controversies surrounding SubQ?
Despite these impressive results, there are several concerns in the industry:
1. Reuse of open-source parameters: SubQ does not use entirely new training data but leverages the parameters from the Chinese open-source model Qwen. Some question whether its success is due to the advanced sparse attention mechanism or the strength of Qwen itself.
2. Limited availability: SubQ is currently only available for a small number of users, preventing widespread validation of its performance. Could this mean that the results seen in laboratory tests are not representative of real-world usage?
These controversies cast some doubt on the significance of SubQ's breakthrough and require more public tests to confirm its capabilities.
V. What changes might this innovation bring about?
If SubQ's technology matures, it could revolutionize the way we process long texts:
1. Replace RAG methods: Current approaches that involve splitting documents into smaller parts for retrieval often miss important information or distort the original logic. SubQ can handle entire documents (e.g., 300-page merger agreements or 1,000 code files) without any loss of context.
2. Reduce costs: Analyzing tens of millions of words used to cost thousands of dollars; now, it might only cost a few dollars, making it accessible to small and medium-sized businesses.
3. Impact on model architecture: Similar to how Transformers replaced recurrent neural networks in 2017, SubQ's dynamic sparse attention could become the standard for future large language models.
Of course, all this depends on whether SubQ can resolve these controversies and whether giants like OpenAI and Google have similar technologies in their arsenal. Nevertheless, it opens up new possibilities for improving the processing capabilities of large language models.
In summary, SubQ represents a significant milestone in the development of large language models. If it proves to be reliable, handling long texts and performing cross-document analyses with AI will become as effortless as having conversations today. However, more validation is needed before we can fully embrace its potential. Let's wait and see.