International Symposium on Grids & Clouds 2020 (ISGC 2020)

The use of HPC at the Tokyo regional analysis center

25 Aug 2020, 14:30

30m

Conference Room 1 (BHSS, Academia Sinica)

Oral Presentation Physics (including HEP) and Engineering Applications Physics & Engineering

Speaker

Dr Michiru Kaneda (ICEPP, the University of Tokyo)

Description

The ATLAS experiment is one of the experiments at the Large Hadron Collider (LHC). It discovered the Higgs boson in 2012 and continues to investigate unsolved physics problems. The current total data amount of ATLAS is more than 200 PB. To manage and process the data, the LHC experiments use the Worldwide LHC Computing Grid (WLCG) project, which is the collaboration of computing centers of 42 countries. Our computing center, the Tokyo regional analysis center, at the International Center for Elementary Particle Physics (ICEPP), the University of Tokyo, is one of the centers of WLCG and it supports ATLAS virtual organization. The system consists of 10,000 CPU cores and 16 PB disk storage. We provide 7,700 CPU cores and 10 PB disk storage to WLCG and remaining resources are dedicated to the local usage of the ATLAS Japan member. All hardware devices are supplied by the three years rental. The current system is the 5th system which started in January 2020. LHC will be upgraded to increase the luminosity 5 times in 2026. The data amount will be increased more than 10 times if we use the current analysis model and both calculation power and storage size will be in short supply. To solve this problem, a lot of software improvements have been done and new analysis models have been developed. However, there is still a gap between the amount of required resources and what we can obtain. Therefore, the WLCG system itself is needed to be upgraded. High-Performance Computing (HPC) resources will be one of the solutions to increase the calculation power of WLCG. In recent years, many countries try to develop new HPCs. The University of Tokyo has several HPC systems, too. We used one of them, so-called Reedbush,and developed the interface between WLCG and Reedbush at the Tokyo regional analysis center. Reedbush system consists of Intel Xeon E5-2695v4 CPU and NVIDIA Tesla P100 GPU. Each node has 2 CPU (36 cores) and 256 GB memory. Some nodes are provided without GPU. All nodes have no direct external network access, which requires special setup to manage input and output data of WLCG jobs. We will talk about how we have established the interface to the HPC in the ATLAS workflow and the experiences in the development.