International Symposium on Grids & Clouds 2020 (ISGC 2020)

SuperBus: Multi-Datacenter Information Service for High Performance Computing Environment Based on Message Bus

Not scheduled

15m

BHSS, Academia Sinica

Poster Presentation Virtual Reserach Environment (including Middleware, tools, services, workflow, … etc.)

Speaker

Ms Can Wu (Computer Network Information Center, Chinese Academy of Sciences)

Description

As the third paradigm of the scientific research, High Performance Computing is widely used in many fields such as computational chemistry, material science, large-scale molecular simulations, etc. High Performance Computing Environment, which aggregates many advanced world-class supercomputers, has received major interest from both academia and industry over the past several years and has become an important platform for scientists and engineers to do scientific research and develop new equipment. With the continuous integration of big data technology, artificial intelligence technology and High Performance Computing technology, there are more and more large-scale computing tasks need to be finished on High Performance Computing Environment. In the meantime, the supercomputer manufacturing develops rapidly. There are more and more better-performing supercomputers with peak performance over 125 PFLOPS joining the High Performance Computing Environment, which poses a huge challenge to the current single-datacenter information service for High Performance Computing Environment. The single-datacenter information service has already encountered performance bottleneck. To improve the performance of the information service for High Performance Computing Environment, this paper proposes an optimized information service called SuperBus based on message bus, which is a multi-datacenter information service. SuperBus is an efficient, scalable and reliable information service. The flexible use of multithread in the efficient data consistency design improves SuperBus’s efficiency significantly; the scalability design makes the system much easier to scale out; the System Security Mechanism and Fault Tolerance Mechanism in the reliability design make the system become more reliable. The experimental results show that the throughput of SuperBus has reached 51000 requests per second, which is extremely efficient. In particular, SuperBus could reduce information transfer time cost, shorten user response time and reach eventual consistency at lower overhead and finally offer users better experience.