21-25 March 2022
Academia Sinica
Europe/Zurich timezone

Providing secure Interactive access to HTCondor batch resources with Jupyterlab

25 Mar 2022, 11:40
20m
Room 1

Room 1

Oral Presentation Track 8: Infrastructure Clouds and Virtualizations Infrastructure Clouds and Virtualisation

Speaker

Carmelo Pellegrino (INFN-CNAF)

Description

The Italian WLCG Tier-1 located in Bologna and managed by INFN provides batch computing resources to several research communities in the fields of High-Energy Physics, Astroparticle Physics, Gravitational Waves, Nuclear Physics and others. The capability of manually executing jobs in the Computing Nodes managed by the Batch System, as they normally would when using owned or cloud resources, is frequently requested by members of our user communities. In fact, the setup-tuning and troubleshooting of a computation campaign on a Batch System can be very difficult and time consuming as on the final Compute Node is never accessible by the end user and because job execution is generally delayed in time with respect to submission. In order to help speeding up the process of setting up the proper execution environment, most batch systems typically support interactive job submission modes, which allow users to execute arbitrary commands in the very same environment where a batch job would run. This is however usually forbidden by administrators for security and other reasons. To overcome these problems and enable users to better and more easily set up their computing tasks, while not impacting on the security policies of the Centre, a new way to deliver interactive on demand access to batch resources has been designed and implemented. This method, based on JupyterHub and JupyterLab technologies, allows users to gain interactive access in a HTCondor job slot executed on the same Compute Nodes running normal batch payloads. This approach allows user groups to address their computing problems in a timely and more effective manner, as the interactive resource being used is the very same where a normal batch job would run. Moreover, the accounting for the used batch resource is the same of a normal job and only based on the lifetime of the user session, which is then accounted as a normal job by the batch system. This work describes the project, the technical details of its implementations, also focusing on the security aspects, the early results and its possible evolution.

Primary authors

Carmelo Pellegrino (INFN-CNAF) Stefano Dal Pra (Istituto Nazionale di Fisica Nucleare)

Presentation materials