Summary of Key Points
As the world's fourth-generation sky survey facilities, such as China's CSST (the "Chinese Hubble"), push astronomical exploration into the era of big data with volumes reaching 100PB (approximately 100 million GB), Chinese scientists are attempting to establish a "centralized AI hub" to address the issues of data dispersion and low utilization. This hub aims to cross-reference data from different telescopes and conduct in-depth analysis to facilitate new scientific discoveries. Additionally, the extreme conditions of astronomical research (high altitudes and space) can serve as a "stress test bed" for domestic hardware technologies, such as chips and storage systems, thereby driving technological breakthroughs. However, the practical challenges of data unification still need to be overcome through the use of unified interfaces and common frameworks.
1. The Era of Astronomical Big Data Has Arrived, but Dispersed Data Has Only Been Scratched on the Surface
Astronomical research today is akin to mining for information—telescopes (like CSST) act as the mining tools, producing vast amounts of data that contain insights into the origins and evolution of the universe. Unfortunately, this data is currently scattered among various teams and is often archived after being used only once, which is like only mining the surface of the mineral deposit.
For instance, a fourth-generation sky survey telescope like CSST generates data equivalent to 100PB (100 million GB of data, roughly equivalent to 100 million 1GB movies). Processing this data requires meticulous cleaning and calibration, as even minor algorithmic errors can be significantly amplified on a large scale. More importantly, many scientific discoveries are made through the cross-comparison of data from different sources (such as combining optical and infrared data). With dispersed data, this process is not feasible, resulting in a waste of the investment made over the years in building the telescopes.
2. The Centralized AI Hub: Bringing Dormant Data to Life
The "centralized AI hub" that scientists envision does not involve simply accumulating all raw data in one place (given its massive volume and the sensitivity of some of the data). Instead, it uses unified interfaces and protocols to connect the dispersed data into an interactive network. For example, the optical data from Telescope A and the infrared data from Telescope B could previously be analyzed separately, but with an AI hub, they can be integrated within the same framework. Scientists can then use this combined data to ask different questions—such as first locating a galaxy and then examining its spectral changes—potentially revealing previously undiscovered patterns. Associate Professor Mao Junjie from Tsinghua University emphasizes that this cross-referencing of data is itself a source of new discoveries.
3. Unifying Data Is Difficult? Start with "Universal Interfaces" and "Common Frameworks"
The data processing methods vary greatly between different telescopes (some measure optical properties, while others measure radio signals, and the instruments and chips used are also different), making it unrealistic to centralize all the raw data directly. The industry's approach is to identify common elements first. For example, the standard processes for cleaning and calibrating optical data are similar across telescopes. These common components can be packaged into a unified framework, and then personalized optimizations can be made for each telescope. Additionally, all parties involved need to agree on using a standard set of interfaces and protocols to enable data to communicate effectively without the need to transfer massive amounts of data (which is both impractical and inefficient). For instance, the person in charge of the Mozi Sky Survey Telescope has stated that if CSST's AI system proves effective, they would be willing to adopt the same platform.
3. The Extreme Conditions of Astronomy: The Ultimate Testing Ground for Hardware Technologies
The environment in which astronomical research is conducted is extremely challenging—telescopes in cold lakes in Qinghai operate at altitudes of 4,300 meters, facing low temperatures and lack of oxygen; space-based telescopes must withstand vacuum, strong radiation, and temperature fluctuations of several hundred degrees Celsius. These conditions provide an ideal environment to test the limits of hardware technologies:
- Storage: The processing and retrieval of 100PB of data require extremely high throughput and speed, which can expose any deficiencies in the storage systems.
- Chips: Space radiation can interfere with chip components, and it is uncertain whether GPUs designed for high altitudes can function properly.
- Technological Extensions: Technologies developed for astronomical applications, such as the giant steel cables used in the FAST (Five-hundred-meter Aperture Spherical Radio Telescope) in China, have been adapted for industrial use. Inspur Information has moved its domestic servers and liquid cooling technology to Daocheng at an altitude of 4,410 meters to build the world's highest cosmic ray computing center, utilizing astronomical scenarios to validate these technologies.
In short, astronomy acts as a "stress tester" that helps identify technical weaknesses in the industry and drives improvements in hardware technologies such as chips and storage.
4. A Win-Win Situation for Science and Industry: The Mutual Beneficial Relationship Between Astronomy and Technology
In the past, the Sloan Sky Survey contributed to the development of Microsoft's database technology. Now, Chinese astronomy aims to follow this same path, using the demands of basic scientific research to drive technological breakthroughs in the industry. For example, solving the challenges of astronomical data processing could lead to more stable and efficient domestic AI chips and storage systems. Conversely, advancements in industry technology can support astronomical research, creating a virtuous cycle. This not only helps us gain a competitive edge in cosmic exploration but also enables hardware technologies to move from the laboratory to the market and be applied in various sectors.
5. The Mutual Beneficial Relationship Between Astronomy and the Economy: The "Feedback Loop" of Scientific Research
Astronomy can have a significant impact on the economy. For instance, the Sloan Sky Survey supported the growth of Microsoft's database technology. Similarly, Chinese astronomy hopes to leverage its needs to drive technological advancements in the industry. By addressing the challenges of data processing, we can improve domestic AI and storage systems. Conversely, technological improvements can enhance astronomical research, fostering a mutually beneficial relationship that benefits both science and industry.