Talks | Parallel Computing
2024
Giulio Malenza
Performance portability via C++ PSTL, SYCL, OpenMP, and HIP: the Gaia AVU-GSR case study Miscellaneous
IEEE, 2024.
Links | BibTeX | Tags: eupex, icsc
@misc{24:malenza:p3hpc,
title = {Performance portability via C++ PSTL, SYCL, OpenMP, and HIP: the Gaia AVU-GSR case study},
author = {Giulio Malenza},
url = {https://datacloud.di.unito.it/index.php/s/55T6LbGcWtssNno},
year = {2024},
date = {2024-11-01},
address = {Atlanta, US},
howpublished = {IEEE},
keywords = {eupex, icsc},
pubstate = {published},
tppubtype = {misc}
}
Marco Edoardo Santimaria, Iacopo Colonnelli, Marco Aldinucci
Releasing the CAPIO middleware from MPI derived constraints Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: across, admire, capio, capiocl, eupex, icsc
@misc{24:santimaria:bighpc,
title = {Releasing the CAPIO middleware from MPI derived constraints},
author = {Marco Edoardo Santimaria and Iacopo Colonnelli and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/zrJGD4i36fWdp5g},
year = {2024},
date = {2024-09-01},
address = {Pisa, Italy},
abstract = {CAPIO is a middleware that transparently injects streaming capabilities into file-based workflows. However, its implementation is limited to HPC environments based on the MPI framework, significantly limiting its applications. This paper will illustrate a proposed architecture and some preliminary results aimed at investigating the usage of a distributed files system as a communication media for the CAPIO middleware, with the ultimate goal of supporting both CLOUD-based and HPC-based workflows.},
keywords = {across, admire, capio, capiocl, eupex, icsc},
pubstate = {published},
tppubtype = {misc}
}
CAPIO is a middleware that transparently injects streaming capabilities into file-based workflows. However, its implementation is limited to HPC environments based on the MPI framework, significantly limiting its applications. This paper will illustrate a proposed architecture and some preliminary results aimed at investigating the usage of a distributed files system as a communication media for the CAPIO middleware, with the ultimate goal of supporting both CLOUD-based and HPC-based workflows.
Marco Edoardo Santimaria, Iacopo Colonnelli, Massimo Torquati, Marco Aldinucci
CAPIO: Cross Application Programamble IO Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: across, admire, capio, capiocl, eupex, icsc
@misc{24:santimaria:itadata:shpcpee,
title = {CAPIO: Cross Application Programamble IO},
author = {Marco Edoardo Santimaria and Iacopo Colonnelli and Massimo Torquati and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/rg6LWwrZXi6tTXm},
year = {2024},
date = {2024-09-01},
address = {Pisa, Italy},
abstract = {With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to expand, further exacerbating quickly the performance gap between computing, memory, and storage technologies. CAPIO (Cross-Application Programmable I/O), is a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. In this presentation, we will introduce the CAPIO-CL language with its semantics, as well as the implementation of the CAPIO-CL language through the CAPIO middleware. We will also provide some case studies of how CAPIO has been employed to improve workflow execution time as well as some future directions.},
keywords = {across, admire, capio, capiocl, eupex, icsc},
pubstate = {published},
tppubtype = {misc}
}
With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to expand, further exacerbating quickly the performance gap between computing, memory, and storage technologies. CAPIO (Cross-Application Programmable I/O), is a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. In this presentation, we will introduce the CAPIO-CL language with its semantics, as well as the implementation of the CAPIO-CL language through the CAPIO middleware. We will also provide some case studies of how CAPIO has been employed to improve workflow execution time as well as some future directions.
Alberto Mulone, Doriana Medić, Marco Aldinucci
A Fault Tolerance mechanism for Hybrid Scientific Workflows Miscellaneous
1st workshop about High-Performance e-Science (HiPES), 2024.
Abstract | Links | BibTeX | Tags: eupex, icsc, streamflow
@misc{24:madrid:hipes:talk,
title = {A Fault Tolerance mechanism for Hybrid Scientific Workflows},
author = {Alberto Mulone and Doriana Medić and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/9Ddj6fGgmDbLDXj},
year = {2024},
date = {2024-08-01},
address = {Madrid, Spain},
abstract = {In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.},
howpublished = {1st workshop about High-Performance e-Science (HiPES)},
keywords = {eupex, icsc, streamflow},
pubstate = {published},
tppubtype = {misc}
}
In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.
Marco Edoardo Santimaria
CAPIO-CL: Cross Application Programmable IO - Coordination Language Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: across, admire, capio, eupex, icsc
@misc{24:santimaria:hlpp:capiocl,
title = {CAPIO-CL: Cross Application Programmable IO - Coordination Language},
author = {Marco Edoardo Santimaria},
url = {https://datacloud.di.unito.it/index.php/s/zsKY3PWzX5NFCiX},
year = {2024},
date = {2024-07-01},
address = {Pisa, Italy},
abstract = {The performance bottleneck in file-based workflows remains a pressing issue in the realm of I/O-based workflows. To address this challenge, a novel annotation language has been developed. CAPIO-CL is positioned as an innovative I/O coordination language, enabling users to annotate data dependencies within file-based workflows with synchronization semantics pertinent to the involved files and directories. Through the information provided by the language, optimization opportunities arise in streaming and preemptive data movement. This paper serves to illustrate the semantics and syntax enabling CAPIO-CL to enhance the performance of in situ workflows without necessitating the rewriting or modification of the original workflow application steps. Finally, an analysis of CAPIO-CL is provided, taking into consideration both language expressiveness and application performance enhancement.},
keywords = {across, admire, capio, eupex, icsc},
pubstate = {published},
tppubtype = {misc}
}
The performance bottleneck in file-based workflows remains a pressing issue in the realm of I/O-based workflows. To address this challenge, a novel annotation language has been developed. CAPIO-CL is positioned as an innovative I/O coordination language, enabling users to annotate data dependencies within file-based workflows with synchronization semantics pertinent to the involved files and directories. Through the information provided by the language, optimization opportunities arise in streaming and preemptive data movement. This paper serves to illustrate the semantics and syntax enabling CAPIO-CL to enhance the performance of in situ workflows without necessitating the rewriting or modification of the original workflow application steps. Finally, an analysis of CAPIO-CL is provided, taking into consideration both language expressiveness and application performance enhancement.
Iacopo Colonnelli, Robert Birke, Giulio Malenza, Gianluca Mittone, Alberto Mulone, Marco Aldinucci
Cross-Facility Federated Learning - Part II Miscellaneous
2024, (Invited talk).
Links | BibTeX | Tags: eupex, icsc, space
@misc{24:ic:elise:xffl,
title = {Cross-Facility Federated Learning - Part II},
author = {Iacopo Colonnelli and Robert Birke and Giulio Malenza and Gianluca Mittone and Alberto Mulone and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/7HonBpcWPxotXLX},
year = {2024},
date = {2024-06-01},
address = {Helsinki, Finland},
note = {Invited talk},
keywords = {eupex, icsc, space},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
CWL in the HPC Ecosystem Miscellaneous
Workshop on workflow languages for HEP analysis, 2024.
Links | BibTeX | Tags: across, eupex, icsc, space, streamflow
@misc{24:icolonne:cwl4hpccern,
title = {CWL in the HPC Ecosystem},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/PRmqdwWHt6P2PH7},
year = {2024},
date = {2024-04-01},
address = {CERN, Meyrin, Switzerland},
howpublished = {Workshop on workflow languages for HEP analysis},
keywords = {across, eupex, icsc, space, streamflow},
pubstate = {published},
tppubtype = {misc}
}
2023
Iacopo Colonnelli, Robert Birke, Giulio Malenza, Gianluca Mittone, Alberto Mulone, Marco Aldinucci, Valerio Basile, Marco Antonio Stranisci, Viviana Patti, Jeroen Galjaard, Lydia Y. Chen, Sanzio Bassini, Massimiliano Guarrasi, Gabriella Scipione, Jan Martinovič, Vit Vondrák
Cross-Facility Federated Learning Miscellaneous
1st EuroHPC User Day, 2023.
Links | BibTeX | Tags: across, ai, eupex, eupilot, HPC
@misc{23:eurohpc,
title = {Cross-Facility Federated Learning},
author = {Iacopo Colonnelli and Robert Birke and Giulio Malenza and Gianluca Mittone and Alberto Mulone and Marco Aldinucci and Valerio Basile and Marco Antonio Stranisci and Viviana Patti and Jeroen Galjaard and Lydia Y. Chen and Sanzio Bassini and Massimiliano Guarrasi and Gabriella Scipione and Jan Martinovič and Vit Vondrák},
url = {https://datacloud.di.unito.it/index.php/s/DDAz4QkJP3WZ68M},
year = {2023},
date = {2023-12-01},
address = {Bruxelles, Belgium},
howpublished = {1st EuroHPC User Day},
keywords = {across, ai, eupex, eupilot, HPC},
pubstate = {published},
tppubtype = {misc}
}
Sofia Karvounari, Eleni Mathioulaki, Michael R. Crusoe, Iacopo Colonnelli
Standardised Workflows at EBRAINS Miscellaneous
Human Brain Project Summit 2023, 2023, (Invited talk).
Abstract | Links | BibTeX | Tags: across, eupex, space, streamflow
@misc{23:HBPSummit,
title = {Standardised Workflows at EBRAINS},
author = {Sofia Karvounari and Eleni Mathioulaki and Michael R. Crusoe and Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/K5YQKTsX9N7NLT8},
year = {2023},
date = {2023-03-01},
address = {Marseille, France},
abstract = {A hands-on training offer for Standardised Workflows in EBRAINS. A short presentation will be used as an introduction, while the main hands-on session will provide information about Writing and Executing Standardised Workflows. TC will give some guidelines, so attendees can experiment with writing CWL tools and workflows and then they will be given access to VM to execute these workflows. The Workflows Dashboard will be also presented during the same session, offering to the attendees the opportunity to understand the different functionalities, use it with TC support and provide useful comments.},
howpublished = {Human Brain Project Summit 2023},
note = {Invited talk},
keywords = {across, eupex, space, streamflow},
pubstate = {published},
tppubtype = {misc}
}
A hands-on training offer for Standardised Workflows in EBRAINS. A short presentation will be used as an introduction, while the main hands-on session will provide information about Writing and Executing Standardised Workflows. TC will give some guidelines, so attendees can experiment with writing CWL tools and workflows and then they will be given access to VM to execute these workflows. The Workflows Dashboard will be also presented during the same session, offering to the attendees the opportunity to understand the different functionalities, use it with TC support and provide useful comments.
Marco Aldinucci
Experimenting with Systems for Decentralized Machine Learning Miscellaneous
NVidia GTC 2023, 2023.
Abstract | Links | BibTeX | Tags: across, admire, epi, eumaster4hpc, eupex, eupilot, hpc4ai, space, textarossa
@misc{23:gtc:fl,
title = {Experimenting with Systems for Decentralized Machine Learning},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/oyLt7xwkbKxz65c},
year = {2023},
date = {2023-03-01},
abstract = {Decentralized machine learning (DML) enables collaborative machine learning without centralized input data. Federated learning (FL) and edge inference (EI) are examples of DML. Collaboration naturally happens at the edge of a distributed system with inherently distributed data. While tools for DML are starting to flourish, much needs to be done to get more flexible and portable tools to experiment with novel techniques, non-fully connected topologies, multiple data domains, and asynchronous collaboration schemes. We'll present recent advances in DML, aiming to improve usability in data centers and, at the edge, to widen the class of models extending FL to non-DNN paradigms, to improve the accuracy of models controlling normalization and frequency of communications, and to boost data privacy though generative adversarial networks. Prerequisites: Intermediate understanding of machine learning methods and distributed & parallel computing.},
howpublished = {NVidia GTC 2023},
keywords = {across, admire, epi, eumaster4hpc, eupex, eupilot, hpc4ai, space, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Decentralized machine learning (DML) enables collaborative machine learning without centralized input data. Federated learning (FL) and edge inference (EI) are examples of DML. Collaboration naturally happens at the edge of a distributed system with inherently distributed data. While tools for DML are starting to flourish, much needs to be done to get more flexible and portable tools to experiment with novel techniques, non-fully connected topologies, multiple data domains, and asynchronous collaboration schemes. We'll present recent advances in DML, aiming to improve usability in data centers and, at the edge, to widen the class of models extending FL to non-DNN paradigms, to improve the accuracy of models controlling normalization and frequency of communications, and to boost data privacy though generative adversarial networks. Prerequisites: Intermediate understanding of machine learning methods and distributed & parallel computing.
Marco Aldinucci
HPC4AI: The Research on AI beyond the public cloud Miscellaneous
CENTAI kick-off meeting, 2023.
Links | BibTeX | Tags: across, admire, brainteaser, epi, eumaster4hpc, eupex, eupilot, hpc4ai, space, textarossa
@misc{23:CENTAI:hpc4ai,
title = {HPC4AI: The Research on AI beyond the public cloud},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/PZXjPm8sfKTmTGb},
year = {2023},
date = {2023-03-01},
address = {Torino, Italy},
howpublished = {CENTAI kick-off meeting},
keywords = {across, admire, brainteaser, epi, eumaster4hpc, eupex, eupilot, hpc4ai, space, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
CWL for HPC: are we there yet? Miscellaneous
2023 CWL Conference, 2023, (Invited talk).
Abstract | Links | BibTeX | Tags: across, eupex, streamflow
@misc{23:CWLConference,
title = {CWL for HPC: are we there yet?},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/CMCd5LiZeXsxwEg},
year = {2023},
date = {2023-03-01},
address = {Heidelberg, Germany},
abstract = {Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWL’s readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.},
howpublished = {2023 CWL Conference},
note = {Invited talk},
keywords = {across, eupex, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWL’s readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.
Marco Aldinucci
From HPC4AI to ICSC living lab: Where systems are the research Miscellaneous
Dell Advanced Computing Workshop 2023: HPC and Beyond, 2023.
Links | BibTeX | Tags: admire, eupex, eupilot, hpc4ai, textarossa
@misc{23:Dell:hpc4ai,
title = {From HPC4AI to ICSC living lab: Where systems are the research},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/M5QRJyDxyxokcfL},
year = {2023},
date = {2023-02-01},
address = {Bologna, Italy},
howpublished = {Dell Advanced Computing Workshop 2023: HPC and Beyond},
keywords = {admire, eupex, eupilot, hpc4ai, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Doriana Medić, Marco Aldinucci
Towards formal model for location aware workflows Miscellaneous
2023.
Abstract | Links | BibTeX | Tags: eupex, icsc
@misc{23:wide:medic,
title = {Towards formal model for location aware workflows},
author = {Doriana Medić and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/wpDd9HETzioixTW},
year = {2023},
date = {2023-01-01},
address = {Torino, Italy},
abstract = {Designing complex applications and executing them on large-scale topologies of heterogeneous architectures is becoming increasingly crucial in many scientific domains. As a result, diverse workflow modelling paradigms are developed, most of them with no formalisation provided. In these circumstances, comparing two different models or switching from one system to the other becomes a hard nut to crack. This paper investigates the capability of process algebra to model a location aware workflow system. Distributed π-calculus is considered as the base of the formal model due to its ability to describe the communicating components that change their structure as an outcome of the communication. Later, it is discussed how the base model could be extended or modified to capture different features of location aware workflow system. The intention of this paper is to highlight the fact that due to its flexibility, π-calculus, could be a good candidate to represent the behavioural perspective of the workflow system.},
keywords = {eupex, icsc},
pubstate = {published},
tppubtype = {misc}
}
Designing complex applications and executing them on large-scale topologies of heterogeneous architectures is becoming increasingly crucial in many scientific domains. As a result, diverse workflow modelling paradigms are developed, most of them with no formalisation provided. In these circumstances, comparing two different models or switching from one system to the other becomes a hard nut to crack. This paper investigates the capability of process algebra to model a location aware workflow system. Distributed π-calculus is considered as the base of the formal model due to its ability to describe the communicating components that change their structure as an outcome of the communication. Later, it is discussed how the base model could be extended or modified to capture different features of location aware workflow system. The intention of this paper is to highlight the fact that due to its flexibility, π-calculus, could be a good candidate to represent the behavioural perspective of the workflow system.
2022
Marco Aldinucci
Il calcolo parallelo: una storia di metodi e algoritmi raccontata dalle macchine Miscellaneous
Olimpiadi di Informatica, 2022, (Invited talk).
Abstract | Links | BibTeX | Tags: across, admire, eumaster4hpc, eupex, eupilot, textarossa
@misc{22:olimpiadi:cs,
title = {Il calcolo parallelo: una storia di metodi e algoritmi raccontata dalle macchine},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/7ZdfLkn3NetzXCN},
year = {2022},
date = {2022-09-01},
address = {Biella, Italy},
abstract = {Lectio Magistralis alle finali nazionali delle Olimpiadi di Informatica 2022},
howpublished = {Olimpiadi di Informatica},
note = {Invited talk},
keywords = {across, admire, eumaster4hpc, eupex, eupilot, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Lectio Magistralis alle finali nazionali delle Olimpiadi di Informatica 2022
Marco Aldinucci
La convergenza HPC-cloud è l'anello mancante tra il calcolo scientifico e l'IA applicata Miscellaneous
Intelligenza Artificiale e Business Applications, 2022, (Invited talk).
Abstract | Links | BibTeX | Tags: across, admire, eumaster4hpc, eupex, eupilot, textarossa
@misc{22:soiel:ai,
title = {La convergenza HPC-cloud è l'anello mancante tra il calcolo scientifico e l'IA applicata},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/xCQSqJ8bCKCXMK9},
year = {2022},
date = {2022-09-01},
address = {Virtual event},
abstract = {Innanzitutto, le infrastrutture HPC stanno adottando le GPU per il loro rapporto prestazioni per watt superiore rispetto ai multicore generici. In secondo luogo, i flussi di lavoro scientifici di prossima generazione stanno integrando passaggi basati sull'intelligenza artificiale per la loro precisione nell'approssimazione e nell'analisi di fenomeni complessi. In terzo luogo, l'IA e in particolare il Machine Learning (ML) rappresentano un carico di lavoro perfetto per le GPU in termini di prestazioni e tempo di sviluppo. Oggi non possiamo ancora chiudere il cerchio eseguendo senza problemi carichi di lavoro scientifici abilitati all'intelligenza artificiale nelle infrastrutture HPC perché il loro software di sistema e gli strumenti di sviluppo non sono progettati per i carichi di lavoro moderni, come i framework ML progettati per il cloud. È probabile che la convergenza HPC-cloud colmi il divario. Nel talk verranno presentate le infrastrutture e gli strumenti sviluppati all'Università di Torino per la convergenza HPC-cloud (es. HPC4AI, StreamFlow, CAPIO, Jupyter-workflow) e come sono stati utilizzati per le applicazioni di intelligenza artificiale, come la diagnosi spiegabile di polmonite COVID-19 e la tutela della privacy AI. L'esperienza maturata nella progettazione e gestione di HPC4AI costituisce il cuore della progettazione del laboratorio di contaminazione del "FutureHPC" di Torino secondo il Centro Nazionale "HPC, BigData e Quantum Computing" finanziato dal PNRR con 320M€ che dovrebbe essere operativo dal 1 settembre 2022. L'obiettivo finale del laboratorio di contaminazione è sviluppare relazioni e collaborazioni tra industria e università.},
howpublished = {Intelligenza Artificiale e Business Applications},
note = {Invited talk},
keywords = {across, admire, eumaster4hpc, eupex, eupilot, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Innanzitutto, le infrastrutture HPC stanno adottando le GPU per il loro rapporto prestazioni per watt superiore rispetto ai multicore generici. In secondo luogo, i flussi di lavoro scientifici di prossima generazione stanno integrando passaggi basati sull'intelligenza artificiale per la loro precisione nell'approssimazione e nell'analisi di fenomeni complessi. In terzo luogo, l'IA e in particolare il Machine Learning (ML) rappresentano un carico di lavoro perfetto per le GPU in termini di prestazioni e tempo di sviluppo. Oggi non possiamo ancora chiudere il cerchio eseguendo senza problemi carichi di lavoro scientifici abilitati all'intelligenza artificiale nelle infrastrutture HPC perché il loro software di sistema e gli strumenti di sviluppo non sono progettati per i carichi di lavoro moderni, come i framework ML progettati per il cloud. È probabile che la convergenza HPC-cloud colmi il divario. Nel talk verranno presentate le infrastrutture e gli strumenti sviluppati all'Università di Torino per la convergenza HPC-cloud (es. HPC4AI, StreamFlow, CAPIO, Jupyter-workflow) e come sono stati utilizzati per le applicazioni di intelligenza artificiale, come la diagnosi spiegabile di polmonite COVID-19 e la tutela della privacy AI. L'esperienza maturata nella progettazione e gestione di HPC4AI costituisce il cuore della progettazione del laboratorio di contaminazione del "FutureHPC" di Torino secondo il Centro Nazionale "HPC, BigData e Quantum Computing" finanziato dal PNRR con 320M€ che dovrebbe essere operativo dal 1 settembre 2022. L'obiettivo finale del laboratorio di contaminazione è sviluppare relazioni e collaborazioni tra industria e università.
Iacopo Colonnelli, Marco Aldinucci
Hybrid Workflows For Large-Scale Scientific Applications Miscellaneous
6th EAGE High Performance Computing Workshop, 2022.
Abstract | Links | BibTeX | Tags: across, eupex, jupyter-workflow, textarossa
@misc{22:eage,
title = {Hybrid Workflows For Large-Scale Scientific Applications},
author = {Iacopo Colonnelli and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/GScPS5LCPdt6Yoo},
year = {2022},
date = {2022-09-01},
address = {Milano, Italy},
abstract = {Large-scale scientific applications are facing an irreversible transition from monolithic, high-performance oriented codes to modular and polyglot deployments of specialised (micro-)services. The reasons behind this transition are many: coupling of standard solvers with Deep Learning techniques, offloading of data analysis and visualisation to Cloud, and the advent of specialised hardware accelerators. Topology-aware Workflow Management Systems (WMSs) play a crucial role. In particular, topology-awareness allows an explicit mapping of workflow steps onto heterogeneous locations, allowing automated executions on top of hybrid architectures (e.g., cloud+HPC or classical+quantum). Plus, topology-aware WMSs can offer non-functional requirements OOTB, e.g. components’ life-cycle orchestration, secure and efficient data transfers, fault tolerance, and cross-cluster execution of urgent workloads. Augmenting interactive Jupyter Notebooks with distributed workflow capabilities allows domain experts to prototype and scale applications using the same technological stack, while relying on a feature-rich and user-friendly web interface. This abstract will showcase how these general methodologies can be applied to a typical geoscience simulation pipeline based on the Full Wavefront Inversion (FWI) technique. In particular, a prototypical Jupyter Notebook will be executed interactively on Cloud. Preliminary data analyses and post-processing will be executed locally, while the computationally demanding optimisation loop will be scheduled on a remote HPC cluster.},
howpublished = {6th EAGE High Performance Computing Workshop},
keywords = {across, eupex, jupyter-workflow, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Large-scale scientific applications are facing an irreversible transition from monolithic, high-performance oriented codes to modular and polyglot deployments of specialised (micro-)services. The reasons behind this transition are many: coupling of standard solvers with Deep Learning techniques, offloading of data analysis and visualisation to Cloud, and the advent of specialised hardware accelerators. Topology-aware Workflow Management Systems (WMSs) play a crucial role. In particular, topology-awareness allows an explicit mapping of workflow steps onto heterogeneous locations, allowing automated executions on top of hybrid architectures (e.g., cloud+HPC or classical+quantum). Plus, topology-aware WMSs can offer non-functional requirements OOTB, e.g. components’ life-cycle orchestration, secure and efficient data transfers, fault tolerance, and cross-cluster execution of urgent workloads. Augmenting interactive Jupyter Notebooks with distributed workflow capabilities allows domain experts to prototype and scale applications using the same technological stack, while relying on a feature-rich and user-friendly web interface. This abstract will showcase how these general methodologies can be applied to a typical geoscience simulation pipeline based on the Full Wavefront Inversion (FWI) technique. In particular, a prototypical Jupyter Notebook will be executed interactively on Cloud. Preliminary data analyses and post-processing will be executed locally, while the computationally demanding optimisation loop will be scheduled on a remote HPC cluster.
Iacopo Colonnelli, Barbara Cantalupo, Doriana Medić, Marco Aldinucci
Hybrid workflows for heterogeneous distributed computing Miscellaneous
3rd Italian Workshop on HPC (ITWSHPC), 2022.
Links | BibTeX | Tags: across, admire, eumaster4hpc, eupex, eupilot, textarossa
@misc{22:itwshpc,
title = {Hybrid workflows for heterogeneous distributed computing},
author = {Iacopo Colonnelli and Barbara Cantalupo and Doriana Medić and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/ienbcA2DJ26aioE},
year = {2022},
date = {2022-09-01},
address = {Torino, Italy},
howpublished = {3rd Italian Workshop on HPC (ITWSHPC)},
keywords = {across, admire, eumaster4hpc, eupex, eupilot, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli, Marco Aldinucci
CINI HPC-KTT: HPC Key Technologies and Tools National Lab Miscellaneous
NVIDIA HPC Roundtable, 2022, (Invited talk).
Links | BibTeX | Tags: across, admire, eumaster4hpc, eupex, eupilot, textarossa
@misc{22:nvidia_hpc_roundtable,
title = {CINI HPC-KTT: HPC Key Technologies and Tools National Lab},
author = {Iacopo Colonnelli and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/9EQniZ2dGzdJ26f},
year = {2022},
date = {2022-09-01},
address = {Casalecchio di Reno, Italy},
howpublished = {NVIDIA HPC Roundtable},
note = {Invited talk},
keywords = {across, admire, eumaster4hpc, eupex, eupilot, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
StreamFlow Miscellaneous
2nd HealthyCloud Workshop: Analysis of existing orchestration mechanisms for distributed computational analyses, 2022, (Invited talk).
Links | BibTeX | Tags: across, deephealth, eupex, streamflow, textarossa
@misc{22:healthycloud-workshop,
title = {StreamFlow},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/Taz8qtzmkmn9ffT},
year = {2022},
date = {2022-07-01},
address = {Virtual event},
howpublished = {2nd HealthyCloud Workshop: Analysis of existing orchestration mechanisms for distributed computational analyses},
note = {Invited talk},
keywords = {across, deephealth, eupex, streamflow, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Marco Aldinucci
EuroHPC and the Italian HPC ecosystem Miscellaneous
Critical Infrastructure Protection Forum - EuroCC Romania, 2022, (Invited talk).
Abstract | Links | BibTeX | Tags: across, admire, eumaster4hpc, eupex, eupilot, icsc, textarossa
@misc{22:cip:romania,
title = {EuroHPC and the Italian HPC ecosystem},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/5dFFoNsZzwTzQkn},
year = {2022},
date = {2022-06-01},
address = {Bucharest, Romania},
abstract = {The talk presents the main investments currently ongoing in Italy in the HPC area as well as the activity of Italian stakeholders within EuroHPC. The novel Italian National Centre on HPC (ICSC) is introduced.},
howpublished = {Critical Infrastructure Protection Forum - EuroCC Romania},
note = {Invited talk},
keywords = {across, admire, eumaster4hpc, eupex, eupilot, icsc, textarossa},
pubstate = {published},
tppubtype = {misc}
}
The talk presents the main investments currently ongoing in Italy in the HPC area as well as the activity of Italian stakeholders within EuroHPC. The novel Italian National Centre on HPC (ICSC) is introduced.
Marco Aldinucci
The Italian HPC ecosystem and the next generation of EuroHPC CoE Miscellaneous
EuroHPC EoCoE final summit, 2022, (Invited talk).
Abstract | Links | BibTeX | Tags: across, admire, eumaster4hpc, eupex, eupilot, icsc, textarossa
@misc{22:eocoe:summit,
title = {The Italian HPC ecosystem and the next generation of EuroHPC CoE},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/AH5Ms3NekeoEooB},
year = {2022},
date = {2022-06-01},
address = {Napoli, Italy},
abstract = {The talk presents the main investments currently ongoing in Italy in the HPC area as well as the activity of Italian stakeholders within EuroHPC. The novel Italian National Centre on HPC (ICSC) is introduced.},
howpublished = {EuroHPC EoCoE final summit},
note = {Invited talk},
keywords = {across, admire, eumaster4hpc, eupex, eupilot, icsc, textarossa},
pubstate = {published},
tppubtype = {misc}
}
The talk presents the main investments currently ongoing in Italy in the HPC area as well as the activity of Italian stakeholders within EuroHPC. The novel Italian National Centre on HPC (ICSC) is introduced.
Iacopo Colonnelli
StreamFlow: a topology-aware WMS Miscellaneous
ELIXIR Cloud, Data & AAI Bi-weekly Technical Calls, 2022, (Invited talk).
Links | BibTeX | Tags: across, deephealth, eupex, streamflow, textarossa
@misc{22:elixir-streamflow,
title = {StreamFlow: a topology-aware WMS},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/Z9GsKnRCxmBdMd3},
year = {2022},
date = {2022-06-01},
address = {Virtual event},
howpublished = {ELIXIR Cloud, Data & AAI Bi-weekly Technical Calls},
note = {Invited talk},
keywords = {across, deephealth, eupex, streamflow, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Marco Aldinucci
From small files to no files Miscellaneous
6th Workshop on Performance and Scalability of Storage Systems, 2022, (Invited talk).
Abstract | Links | BibTeX | Tags: admire, eupex
@misc{22:p3s:capio,
title = {From small files to no files},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/KLDi87xQmX86iXg},
year = {2022},
date = {2022-06-01},
address = {Paris, France},
abstract = {Modern distributed high-performance storage systems saturate the network bandwidth, and the margins for improvement at the software level are tiny. Due to metadata access, they might be troubled with massive access to small files. An example is the Software Heritage (SH) dataset, half petabytes of files with an average size of 3kBytes (Terabytes of metadata). While working with SH, we developed the idea of substituting files with in-memory streams. We did it living in dread with the fear of asking application programmers to rewrite their lovely antique legacy code exploiting the POSIX interface, and up to now, we did not. In the talk, we will introduce CAPIO (Cross-Application Programmable I/O) design principles and the current state of development of the prototype.},
howpublished = {6th Workshop on Performance and Scalability of Storage Systems},
note = {Invited talk},
keywords = {admire, eupex},
pubstate = {published},
tppubtype = {misc}
}
Modern distributed high-performance storage systems saturate the network bandwidth, and the margins for improvement at the software level are tiny. Due to metadata access, they might be troubled with massive access to small files. An example is the Software Heritage (SH) dataset, half petabytes of files with an average size of 3kBytes (Terabytes of metadata). While working with SH, we developed the idea of substituting files with in-memory streams. We did it living in dread with the fear of asking application programmers to rewrite their lovely antique legacy code exploiting the POSIX interface, and up to now, we did not. In the talk, we will introduce CAPIO (Cross-Application Programmable I/O) design principles and the current state of development of the prototype.
Marco Aldinucci
HPC-cloud convergence is the missing link between scientific computing and applied-AI Miscellaneous
Machine Learning for Astrophysics (ML4ASTRO), 2022, (Keynote talk).
Abstract | Links | BibTeX | Tags: across, deephealth, eupex, eupilot
@misc{22:ml4astrotalk,
title = {HPC-cloud convergence is the missing link between scientific computing and applied-AI},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/2SGswkcip7MoMoH},
year = {2022},
date = {2022-06-01},
address = {Catania, Italy},
abstract = {First, HPC infrastructures are embracing GPUs for their superior performance-per-watt ratio against general-purpose multicores. Second, the next-generation scientific workflows are integrating AI-based steps for their accuracy in approximating and analyzing complex phenomena. Third, AI and specifically Machine Learning (ML), is a perfect workload for GPUs in terms of performance and development time. Today, we cannot still close the circle seamlessly running AI-enabled scientific workloads into HPC infrastructures because their system software and development tools are not designed for modern workloads, such as ML frameworks designed for the cloud. HPC-cloud convergence is likely to bridge the gap. In the talk, we will present Streamflow and CAPIO, two development tools for HPC-cloud convergence.},
howpublished = {Machine Learning for Astrophysics (ML4ASTRO)},
note = {Keynote talk},
keywords = {across, deephealth, eupex, eupilot},
pubstate = {published},
tppubtype = {misc}
}
First, HPC infrastructures are embracing GPUs for their superior performance-per-watt ratio against general-purpose multicores. Second, the next-generation scientific workflows are integrating AI-based steps for their accuracy in approximating and analyzing complex phenomena. Third, AI and specifically Machine Learning (ML), is a perfect workload for GPUs in terms of performance and development time. Today, we cannot still close the circle seamlessly running AI-enabled scientific workloads into HPC infrastructures because their system software and development tools are not designed for modern workloads, such as ML frameworks designed for the cloud. HPC-cloud convergence is likely to bridge the gap. In the talk, we will present Streamflow and CAPIO, two development tools for HPC-cloud convergence.
Marco Aldinucci
Cognitive continuum: a game theoretical approach Miscellaneous
HiPEAC Vision meeting, Brussels, 16 May 2022, 2022.
Abstract | Links | BibTeX | Tags: across, admire, brainteaser, eumaster4hpc, eupex, eupilot, textarossa
@misc{22:hipeacvision:fl,
title = {Cognitive continuum: a game theoretical approach},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/453HWfmrQyo7j9E},
year = {2022},
date = {2022-05-01},
address = {Brussels, Belgium},
abstract = {Cognitive continuum: a game theoretical approach, (maybe) data operations are too basic: read, write, copy, remove … The talk is aimed to contribute to the forthcoming HiPEAC Vision document},
howpublished = {HiPEAC Vision meeting, Brussels, 16 May 2022},
keywords = {across, admire, brainteaser, eumaster4hpc, eupex, eupilot, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Cognitive continuum: a game theoretical approach, (maybe) data operations are too basic: read, write, copy, remove … The talk is aimed to contribute to the forthcoming HiPEAC Vision document
Marco Aldinucci
Da HPC4AI al living lab dello spoke FutureHPC del Centro Nazionale HPC Miscellaneous
Condivisioni, Conferenza GARR 2022, 2022, (Keynote talk).
Abstract | Links | BibTeX | Tags: across, admire, eumaster4hpc, eupex, hpc4ai, icsc, textarossa
@misc{22:garr,
title = {Da HPC4AI al living lab dello spoke FutureHPC del Centro Nazionale HPC},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/P3KSroSSmrRxZMc},
year = {2022},
date = {2022-05-01},
address = {Palermo, Italy},
abstract = {HPC4AI is an open-access laboratory of the University of Turin open to researchers, students and companies that manages a double pair of systems: a production cloud-HPC system and its twin dedicated to development. The cloud-HPC system is implemented thanks to an extended version of the GARR cloud (OpenStack) and the SLURM workload manager. HPC4AI is specifically designed to support system software development and cloud-HPC convergence tools. Among these streamflow (WMS), jupyter-as-a-service (SaaS), portable-secure-tenant (PasS). The experience gained in the design and management of HPC4AI forms the heart of the design of the livinglab of the Turin "FutureHPC" spoke of the National Center "HPC, BigData and Quantum Computing" funded by the PNRR which should be operational from September 2022.},
howpublished = {Condivisioni, Conferenza GARR 2022},
note = {Keynote talk},
keywords = {across, admire, eumaster4hpc, eupex, hpc4ai, icsc, textarossa},
pubstate = {published},
tppubtype = {misc}
}
HPC4AI is an open-access laboratory of the University of Turin open to researchers, students and companies that manages a double pair of systems: a production cloud-HPC system and its twin dedicated to development. The cloud-HPC system is implemented thanks to an extended version of the GARR cloud (OpenStack) and the SLURM workload manager. HPC4AI is specifically designed to support system software development and cloud-HPC convergence tools. Among these streamflow (WMS), jupyter-as-a-service (SaaS), portable-secure-tenant (PasS). The experience gained in the design and management of HPC4AI forms the heart of the design of the livinglab of the Turin "FutureHPC" spoke of the National Center "HPC, BigData and Quantum Computing" funded by the PNRR which should be operational from September 2022.
Iacopo Colonnelli
StreamFlow: A framework for hybrid workflows Miscellaneous
EUPEX WP5 bi-weekly meeting, 2022.
Links | BibTeX | Tags: eupex, streamflow
@misc{22:eupex-streamflow,
title = {StreamFlow: A framework for hybrid workflows},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/NjKEySP7HfrCQHZ},
year = {2022},
date = {2022-04-01},
address = {Virtual event},
howpublished = {EUPEX WP5 bi-weekly meeting},
keywords = {eupex, streamflow},
pubstate = {published},
tppubtype = {misc}
}
2021
Marco Aldinucci
The Italian research on HPC key technologies across EuroHPC Miscellaneous
2021.
Abstract | Links | BibTeX | Tags: across, admire, eupex, eupilot, textarossa
@misc{21:CINI_acm_CF_talk,
title = {The Italian research on HPC key technologies across EuroHPC},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/3ZYmDbEm84rbB9k},
year = {2021},
date = {2021-05-01},
booktitle = {ACM Computing Frontiers},
publisher = {ACM},
address = {Virtual Conference, Italy},
abstract = {High-Performance Computing (HPC) is one of the strategic priorities for research and innovation worldwide due to its relevance for industrial and scientific applications. We envision HPC as composed of three pillars: infrastructures, applications, and key technologies and tools. While infrastructures are by construction centralized in large-scale HPC centers, and applications are generally within the purview of domain-specific organizations, key technologies fall in an intermediate case where coordination is needed, but design and development are often decentralized. A large group of Italian researchers has started a dedicated laboratory within the National Interuniversity Consortium for Informatics (CINI) to address this challenge. The laboratory, albeit young, has managed to succeed in its first attempts to propose a coordinated approach to HPC research within the EuroHPC Joint Undertaking, participating in the calls 2019-20 to five successful proposals for an aggregate total cost of 95M Euro. In this paper, we outline the working group's scope and goals and provide an overview of the five funded projects, which become fully operational in March 2021, and cover a selection of key technologies provided by the working group partners, highlighting their usage development within the projects.},
keywords = {across, admire, eupex, eupilot, textarossa},
pubstate = {published},
tppubtype = {misc}
}
High-Performance Computing (HPC) is one of the strategic priorities for research and innovation worldwide due to its relevance for industrial and scientific applications. We envision HPC as composed of three pillars: infrastructures, applications, and key technologies and tools. While infrastructures are by construction centralized in large-scale HPC centers, and applications are generally within the purview of domain-specific organizations, key technologies fall in an intermediate case where coordination is needed, but design and development are often decentralized. A large group of Italian researchers has started a dedicated laboratory within the National Interuniversity Consortium for Informatics (CINI) to address this challenge. The laboratory, albeit young, has managed to succeed in its first attempts to propose a coordinated approach to HPC research within the EuroHPC Joint Undertaking, participating in the calls 2019-20 to five successful proposals for an aggregate total cost of 95M Euro. In this paper, we outline the working group's scope and goals and provide an overview of the five funded projects, which become fully operational in March 2021, and cover a selection of key technologies provided by the working group partners, highlighting their usage development within the projects.