21-25 March 2022
Academia Sinica
Europe/Zurich timezone

On-demand scheduling of GPUs for CI/CD with Kubernetes on Openstack

25 Mar 2022, 11:00
20m
Room 1

Room 1

Oral Presentation Track 8: Infrastructure Clouds and Virtualizations Infrastructure Clouds and Virtualisation

Speaker

Tim Wetzel (Deutsches Elektronen-Synchrotron DESY)

Description

Machine Learning and Artificial Intelligence tools have become more and more commonplace in research over the last years and a growing need for organising models and source codes emerges. For the latter task, there are several version control tools, of which git is the de-facto standard. Together with Continuous Integration and Continuous Deployment (CI/CD) pipelines, however, git unfolds a lot more potential. We have established a GitLab platform with elastic CI/CD services for our researchers over the last years and now additionally provide GPUs for CI/CD workflows and pipelines for building, performance measurements and regression testing.

We are running an on-premise Openstack IaaS cloud for virtual machine provisioning, which we use to spawn manage Kubernetes clusters as a platform for deploying CI/CD runners. This allows easy scaling and straightforward integration of different hardware components under a common API.
This portability allows to interface with federated resources from large collaborations such as EOSC, EGI, and platforms within the Helmholtz Association (HIFIS, HIP).
In this talk, we will provide detail on our infrastructure setup, and elaborate on demo use cases for training networks with our platform.

Primary authors

Michael Schuh (DESY) Tim Wetzel (Deutsches Elektronen-Synchrotron DESY) Patrick Fuhrmann (DESY/dCache.org)

Co-authors

Presentation materials