International Symposium on Grids & Clouds 2019 (ISGC 2019)

Information Service for High Performance Computing Environment based on Message Bus

5 Apr 2019, 11:20

30m

Media Conference Room (Academia Sinica)

Oral Presentation Virtual Reserach Environment (including Middleware, tools, services, workflow, … etc.) VRE

Speaker

Dr yining zhao (Computer Network Information Center, Chinese Academy of Sciences)

Description

The High Performance Computing Environment in China (a.k.a. China National Grid, or CNGrid) aggregates the majority of China’s supercomputers including Sunway TaihuLight and Milkyway-2 with 200PF aggregated computing resource and 167PB aggregated storage resource, and provides unified and convenient high-performance computing services for users. With the advent of the Big Data Era, the information service in the High Performance Computing Environment is facing a huge challenge. On one hand, the development of high-performance computer manufacturing brings more supercomputers into the High Performance Computing Environment. And a single supercomputer can provide more resource scale and support more application tasks; On the other hand, as the combination of Big Data technology and High Performance Computing technology, more and more Big Data applications are running in High Performance Computing Environment. The amount of resource information is surging, which brings a huge challenge to the information service. SCE is the middleware software in the High Performance Computing Environment and it is developed by Computer Network Information Center, Chinese Academy of Sciences (CAS). After the arrival of the Big Data Era, the resource information in SCE increases rapidly. How to quickly update a huge number of resource information for job scheduling and accurately provide real-time information for users becomes a key problem. To provide efficient and stable information service for users, an optimized information service is necessary. This paper proposes an optimized information service based on message bus, which is efficient, scalable and simple. The optimized information service standardizes resource information which makes it easier to scale out. The message bus uses ZooKeeper cluster and Kafka cluster to transfer information, which improves the throughput greatly and supports multiple management centers to share resource information and to provide information management services at the same time, which reduces the time cost effectively. The security system in message bus is consisted of authority management, identity authentication and data backup, which ensures the system security. The optimized information service improves information update accuracy and security which gives users better experience. As the experimental experiment results shown, the optimized information service provides efficient information service with lower cost and lower system load by standardizing information and transferring information concurrently. Meanwhile, the security system and simple interfaces makes the information service more available and much easier to use. The optimized information service is an efficient, available and simple information service under the Big Data Era environment.