24-29 March 2024
BHSS, Academia Sinica
Asia/Taipei timezone

Enhanced StoRM WebDAV data transfer performance with a new deployment architecture behind NGINX reverse proxy

28 Mar 2024, 14:40
20m
Media Conf. Room (BHSS, Academia Sinica)

Media Conf. Room

BHSS, Academia Sinica

Oral Presentation Track 6: Data Management & Big Data Data Management & Big Data

Speakers

Francesco Giacomini (INFN) Enrico Vianello (INFN)

Description

StoRM WebDAV is a component of StoRM (Storage Resource Manager) which is designed to provide a scalable and efficient solution for managing data storage and access in Grid computing environments. StoRM WebDAV specifically focuses on enabling access to stored data through the WebDAV (Web Distributed Authoring and Versioning) protocol. WebDAV is an extension of the HTTP protocol that allows users to create, change and move resources on a web server.

StoRM WebDAV is designed to follow the requirements set forth by the WLCG (Worldwide LHC Computing Grid) community, in particular it supports: Third Party Copies (TPC), authorization based on JWT tokens or X.509 certificates (often VOMS proxies), and fine-grained access policies. Third-party copy has been one of main GridFTP features used by LHC experiments data management frameworks to implement scalable data transfer and management. In 2017 the Globus Alliance announced that the open-source Globus Toolkit would no longer be supported. This seriously impacted the WLCG community because of the central role of the Globus Security Infrastructure and GridFTP in the context of data transfer frameworks. As a natural consequence, WLCG is moving towards HTTP-based data transfers. In this HTTP-based context, an extension of the WebDAV COPY verb has been defined by the WLCG community and consists of bulk transfer requests between two remote storage endpoints.

This contribution highlights how data transfer (TPC or direct upload/download) performance can be enhanced by delegating them to NGINX. This strategic decision is driven by the proven reliability, scalability, and performance capabilities of NGINX in handling such critical functions.

Primary authors

Presentation materials