Speakers
Daniele Cesini
(INFN-CNAF)
Prof.
Davide Salomoni
(INFN)
Description
In the past year, INFN has been actively working on several projects that involve close collaborations between research and industry. In these collaborations, industries typically bring in complex use cases that involve data collection from heterogeneous sources, often originated by many distributed sensors, while INFN works together with them to define proper architectures to handle these use cases. They normally follow a three-tier approach, with levels such as IoT, Edge and Cloud.
In this talk, we briefly discuss the most challenging use cases, moving on to report on some state-of-the-art solutions we have been developing, involving data ingestion from streamed data in real-time or quasi real-time, processing and analytics.
More in detail, a key point of our architecture is typically the integration of reusable, open components at multiple layers, providing vendor neutrality in terms of infrastructures, with rapid prototyping and deployment.
At the Edge level, we show how data processing and an AI-based inference engine can be implemented through a reusable set of pre-trained neural networks and with use of Function as a Service solutions.
At the Cloud level, we show how our "Infrastructure as Code" approach led us to automatize data transfer, possibly replicating and respecting privacy for data sets where this is necessary, to do continuous training of changing data sets, and to set up repositories for complete infrastructural solutions (e.g. dynamically provisioned Spark clusters or monitoring engines), for AI models, down to executable functions. These components can all be published, used and reused in either testing or production mode, to verify changes before they enter live status, eventually shortening time to market.
We conclude the talk by showing how these solutions are not only important for industry, but are of immediate interest and use also for several scientific applications, and how they have been integrated into the INFN nation-wide distributed infrastructure.
Primary author
Prof.
Davide Salomoni
(INFN)
