Talks | Parallel Computing
2024
Gianluca Mittone, Alberto Mulone, Iacopo Colonnelli, Robert Birke, Marco Aldinucci
Enabling Cross-Facility LLMs Pre-Training Miscellaneous
Accelerating the Development and Use of Generative AI for Science and Engineering: The Trillion Parameter Consortium (TPC), 2024.
Abstract | Links | BibTeX | Tags: eupilot, fl, icsc, space, streamflow
@misc{24:mittone:TPC,
title = {Enabling Cross-Facility LLMs Pre-Training},
author = {Gianluca Mittone and Alberto Mulone and Iacopo Colonnelli and Robert Birke and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/DRgm8ebBkKQgD2d},
year = {2024},
date = {2024-11-01},
address = {Atlanta, GE, USA},
abstract = {Big-tech companies pre-train SOTA LLMs on special-purpose, private HPCs, while public research centres lack the resources to compete. We advocate a new take on large model training, e.g., LLMs, called xFFL, which leverages federated learning as an enabling technique to exploit geographically distributed computing power to bridge such digital divide. This work introduces a proof-of-concept federated training of LLaMA-3 8B on three EuroHPC Top500 facilities, proving the viability of leveraging cross-facility publicly available computational power to sustain SOTA LLM workloads.},
howpublished = {Accelerating the Development and Use of Generative AI for Science and Engineering: The Trillion Parameter Consortium (TPC)},
keywords = {eupilot, fl, icsc, space, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Big-tech companies pre-train SOTA LLMs on special-purpose, private HPCs, while public research centres lack the resources to compete. We advocate a new take on large model training, e.g., LLMs, called xFFL, which leverages federated learning as an enabling technique to exploit geographically distributed computing power to bridge such digital divide. This work introduces a proof-of-concept federated training of LLaMA-3 8B on three EuroHPC Top500 facilities, proving the viability of leveraging cross-facility publicly available computational power to sustain SOTA LLM workloads.
Alberto Mulone, Doriana Medić, Marco Aldinucci
A Fault Tolerance mechanism for Hybrid Scientific Workflows Miscellaneous
1st workshop about High-Performance e-Science (HiPES), 2024.
Abstract | Links | BibTeX | Tags: eupex, icsc, streamflow
@misc{24:madrid:hipes:talk,
title = {A Fault Tolerance mechanism for Hybrid Scientific Workflows},
author = {Alberto Mulone and Doriana Medić and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/9Ddj6fGgmDbLDXj},
year = {2024},
date = {2024-08-01},
address = {Madrid, Spain},
abstract = {In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.},
howpublished = {1st workshop about High-Performance e-Science (HiPES)},
keywords = {eupex, icsc, streamflow},
pubstate = {published},
tppubtype = {misc}
}
In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.
Iacopo Colonnelli
Dynamic hybrid workflows for Deep Learning on HPC infrastructure Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: icsc, jupyter-workflow, streamflow
@misc{24:icolonne:ictp,
title = {Dynamic hybrid workflows for Deep Learning on HPC infrastructure},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/EaFHJEKNbW5oXeq},
year = {2024},
date = {2024-05-01},
address = {Trieste, Italy},
abstract = {Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher’s laptop to an entire HPC facility.},
keywords = {icsc, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher’s laptop to an entire HPC facility.
Iacopo Colonnelli
CWL in the HPC Ecosystem Miscellaneous
Workshop on workflow languages for HEP analysis, 2024.
Links | BibTeX | Tags: across, eupex, icsc, space, streamflow
@misc{24:icolonne:cwl4hpccern,
title = {CWL in the HPC Ecosystem},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/PRmqdwWHt6P2PH7},
year = {2024},
date = {2024-04-01},
address = {CERN, Meyrin, Switzerland},
howpublished = {Workshop on workflow languages for HEP analysis},
keywords = {across, eupex, icsc, space, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Alberto Mulone
Cross-Platform Full Waveform Inversion Miscellaneous
ICSC - Spoke 1 meeting, 2024.
Abstract | Links | BibTeX | Tags: icsc, streamflow
@misc{24:icsc:spoke1:eni,
title = {Cross-Platform Full Waveform Inversion},
author = {Alberto Mulone},
url = {https://datacloud.di.unito.it/index.php/s/M3HkxA5wsBPS5ro},
year = {2024},
date = {2024-02-01},
address = {Torino, Italy},
abstract = {Presentation about the Innovation Grant in collaboration with ENI},
howpublished = {ICSC - Spoke 1 meeting},
keywords = {icsc, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Presentation about the Innovation Grant in collaboration with ENI
2023
Alberto Scionti, Iacopo Colonnelli
Orchestrating Multi-Domain Workflows: The ACROSS Approach Miscellaneous
Workflows Community: Modern Workflows for Continuum and Cross-Facility Computing, 2023.
Links | BibTeX | Tags: across, streamflow
@misc{23:sc:WCIBoF,
title = {Orchestrating Multi-Domain Workflows: The ACROSS Approach},
author = {Alberto Scionti and Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/rJXcDBK4mLmS8yz},
year = {2023},
date = {2023-11-01},
address = {Denver, CO, Usa},
howpublished = {Workflows Community: Modern Workflows for Continuum and Cross-Facility Computing},
keywords = {across, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
ACROSS: HPC Big Data Artificial Intelligence Cross Stack Platform Towards Exascale Miscellaneous
LN HPC-KTT Assemblea Nazionale 2023, 2023.
Links | BibTeX | Tags: across, streamflow
@misc{23:AssembleaHPC-KTT,
title = {ACROSS: HPC Big Data Artificial Intelligence Cross Stack Platform Towards Exascale},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/aK7es8BgFeWorjD},
year = {2023},
date = {2023-10-01},
address = {Pisa, Italy},
howpublished = {LN HPC-KTT Assemblea Nazionale 2023},
keywords = {across, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli, Doriana Medić, Barbara Cantalupo, Marco Aldinucci
Università degli Studi di Torino: Alpha parallel research group Miscellaneous
HaMMon Kick-Off meeting, 2023.
Links | BibTeX | Tags: icsc, streamflow
@misc{23:HaMMonProject,
title = {Università degli Studi di Torino: Alpha parallel research group},
author = {Iacopo Colonnelli and Doriana Medić and Barbara Cantalupo and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/cmgy9BZ3nwCR2QJ},
year = {2023},
date = {2023-10-01},
address = {Bologna, Italy},
howpublished = {HaMMon Kick-Off meeting},
keywords = {icsc, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
Workflow models for heterogeneous distributed systems Miscellaneous
2nd Italian Conference on Big Data and Data Science (ITADATA 2023), 2023, (Best PhD Thesis Award).
Links | BibTeX | Tags: jupyter-workflow, streamflow
@misc{23:ITADATABestPhDThesis,
title = {Workflow models for heterogeneous distributed systems},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/6RqcaJ4djqFNDC8},
year = {2023},
date = {2023-09-01},
address = {Napoli, Italy},
howpublished = {2nd Italian Conference on Big Data and Data Science (ITADATA 2023)},
note = {Best PhD Thesis Award},
keywords = {jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Alberto Mulone, Sherine Awad, Davide Chiarugi, Marco Aldinucci
Porting the Variant Calling Pipeline for NGS data in cloud-HPC environment Miscellaneous
47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023, 2023.
Abstract | Links | BibTeX | Tags: across, icsc, streamflow
@misc{23:mulone:wide:talk,
title = {Porting the Variant Calling Pipeline for NGS data in cloud-HPC environment},
author = {Alberto Mulone and Sherine Awad and Davide Chiarugi and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/zNLj3LCZNsNxHwy},
year = {2023},
date = {2023-06-01},
address = {Torino, Italy},
abstract = {In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.},
howpublished = {47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023},
keywords = {across, icsc, streamflow},
pubstate = {published},
tppubtype = {misc}
}
In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.
Iacopo Colonnelli
Workflows and the Common Workflow Language (CWL) Miscellaneous
OSA2Micro: An Open Science Approach to Microbiology data integration, 2023, (Invited talk).
Links | BibTeX | Tags: streamflow
@misc{23:SA2Micro,
title = {Workflows and the Common Workflow Language (CWL)},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/NHWWzMMaQgAsA52},
year = {2023},
date = {2023-05-01},
address = {Torino, Italy},
howpublished = {OSA2Micro: An Open Science Approach to Microbiology data integration},
note = {Invited talk},
keywords = {streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
UNITO tools presentation Miscellaneous
CN HPC Flagship 3 Working Day, 2023.
Links | BibTeX | Tags: jupyter-workflow, streamflow
@misc{23:FL3WorkingDay,
title = {UNITO tools presentation},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/fgHbnLDQSFtcwLd},
year = {2023},
date = {2023-05-01},
address = {Bologna, Italy},
howpublished = {CN HPC Flagship 3 Working Day},
keywords = {jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Sofia Karvounari, Eleni Mathioulaki, Michael R. Crusoe, Iacopo Colonnelli
Standardised Workflows at EBRAINS Miscellaneous
Human Brain Project Summit 2023, 2023, (Invited talk).
Abstract | Links | BibTeX | Tags: across, eupex, space, streamflow
@misc{23:HBPSummit,
title = {Standardised Workflows at EBRAINS},
author = {Sofia Karvounari and Eleni Mathioulaki and Michael R. Crusoe and Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/K5YQKTsX9N7NLT8},
year = {2023},
date = {2023-03-01},
address = {Marseille, France},
abstract = {A hands-on training offer for Standardised Workflows in EBRAINS. A short presentation will be used as an introduction, while the main hands-on session will provide information about Writing and Executing Standardised Workflows. TC will give some guidelines, so attendees can experiment with writing CWL tools and workflows and then they will be given access to VM to execute these workflows. The Workflows Dashboard will be also presented during the same session, offering to the attendees the opportunity to understand the different functionalities, use it with TC support and provide useful comments.},
howpublished = {Human Brain Project Summit 2023},
note = {Invited talk},
keywords = {across, eupex, space, streamflow},
pubstate = {published},
tppubtype = {misc}
}
A hands-on training offer for Standardised Workflows in EBRAINS. A short presentation will be used as an introduction, while the main hands-on session will provide information about Writing and Executing Standardised Workflows. TC will give some guidelines, so attendees can experiment with writing CWL tools and workflows and then they will be given access to VM to execute these workflows. The Workflows Dashboard will be also presented during the same session, offering to the attendees the opportunity to understand the different functionalities, use it with TC support and provide useful comments.
Iacopo Colonnelli
CWL for HPC: are we there yet? Miscellaneous
2023 CWL Conference, 2023, (Invited talk).
Abstract | Links | BibTeX | Tags: across, eupex, streamflow
@misc{23:CWLConference,
title = {CWL for HPC: are we there yet?},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/CMCd5LiZeXsxwEg},
year = {2023},
date = {2023-03-01},
address = {Heidelberg, Germany},
abstract = {Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWL’s readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.},
howpublished = {2023 CWL Conference},
note = {Invited talk},
keywords = {across, eupex, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWL’s readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.
2022
Iacopo Colonnelli
StreamFlow Miscellaneous
2nd HealthyCloud Workshop: Analysis of existing orchestration mechanisms for distributed computational analyses, 2022, (Invited talk).
Links | BibTeX | Tags: across, deephealth, eupex, streamflow, textarossa
@misc{22:healthycloud-workshop,
title = {StreamFlow},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/Taz8qtzmkmn9ffT},
year = {2022},
date = {2022-07-01},
address = {Virtual event},
howpublished = {2nd HealthyCloud Workshop: Analysis of existing orchestration mechanisms for distributed computational analyses},
note = {Invited talk},
keywords = {across, deephealth, eupex, streamflow, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
StreamFlow: a topology-aware WMS Miscellaneous
ELIXIR Cloud, Data & AAI Bi-weekly Technical Calls, 2022, (Invited talk).
Links | BibTeX | Tags: across, deephealth, eupex, streamflow, textarossa
@misc{22:elixir-streamflow,
title = {StreamFlow: a topology-aware WMS},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/Z9GsKnRCxmBdMd3},
year = {2022},
date = {2022-06-01},
address = {Virtual event},
howpublished = {ELIXIR Cloud, Data & AAI Bi-weekly Technical Calls},
note = {Invited talk},
keywords = {across, deephealth, eupex, streamflow, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
StreamFlow: A framework for hybrid workflows Miscellaneous
EUPEX WP5 bi-weekly meeting, 2022.
Links | BibTeX | Tags: eupex, streamflow
@misc{22:eupex-streamflow,
title = {StreamFlow: A framework for hybrid workflows},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/NjKEySP7HfrCQHZ},
year = {2022},
date = {2022-04-01},
address = {Virtual event},
howpublished = {EUPEX WP5 bi-weekly meeting},
keywords = {eupex, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli, Dario Tranchitella
OpenDeepHealth: Crafting a Deep Learning Platform as a Service with Kubernetes Miscellaneous
J on The Beach 2022, 2022.
Links | BibTeX | Tags: across, deephealth, hpc4ai, jupyter-workflow, streamflow
@misc{22:jotb22,
title = {OpenDeepHealth: Crafting a Deep Learning Platform as a Service with Kubernetes},
author = {Iacopo Colonnelli and Dario Tranchitella},
url = {https://datacloud.di.unito.it/index.php/s/n6J7STNnwdyqtET},
year = {2022},
date = {2022-04-01},
address = {Malaga, Spain},
howpublished = {J on The Beach 2022},
keywords = {across, deephealth, hpc4ai, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
Distributed workflows with Jupyter Miscellaneous
J on The Beach 2022, 2022, (Workshop).
Links | BibTeX | Tags: across, deephealth, jupyter-workflow, streamflow
@misc{22:jotb22-workshop,
title = {Distributed workflows with Jupyter},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/om89q55S6ePf2Ji},
year = {2022},
date = {2022-04-01},
address = {Malaga, Spain},
howpublished = {J on The Beach 2022},
note = {Workshop},
keywords = {across, deephealth, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
StreamFlow: A framework for hybrid workflows Miscellaneous
ACROSS WP4 meeting, 2022.
Links | BibTeX | Tags: across, streamflow
@misc{22:across-streamflow,
title = {StreamFlow: A framework for hybrid workflows},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/FXFTKtQSRf6anMX},
year = {2022},
date = {2022-02-01},
address = {Virtual event},
howpublished = {ACROSS WP4 meeting},
keywords = {across, streamflow},
pubstate = {published},
tppubtype = {misc}
}
2021
Iacopo Colonnelli
StreamFlow: A framework for hybrid workflows Miscellaneous
ACROSS WP4 meeting, 2021.
Links | BibTeX | Tags: across, streamflow
@misc{21:across-streamflow,
title = {StreamFlow: A framework for hybrid workflows},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/yrGYJL6CyNywF8a},
year = {2021},
date = {2021-10-01},
address = {Virtual event},
howpublished = {ACROSS WP4 meeting},
keywords = {across, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Marco Aldinucci
The modernization of HPC applications for the cloud era Miscellaneous
Fifth EAGE Workshop on High Performance Computing for Upstream, 2021, (Keynote talk).
Abstract | BibTeX | Tags: across, admire, deephealth, streamflow
@misc{21:eni:streamflow,
title = {The modernization of HPC applications for the cloud era},
author = {Marco Aldinucci},
year = {2021},
date = {2021-09-01},
address = {Virtual event},
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g., clouds, supercomputers, and both of them. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments (such as Kubernetes and SLURM), making it possible to execute onto multiple sites not sharing a common data space. Streamflow clearly distinguishes it from many other workflow management systems because it decouples the data dependencies from the deployment of (containerized) workflow steps. Streamflow also leverages CAPIO (Cross-Application Programmable I/O) to move data from one step to another efficiently. CAPIO captures the POSIX file system and streams it in parallel and in-memory to the workflow's next step, possibly enabling in-transit data filtering.},
howpublished = {Fifth EAGE Workshop on High Performance Computing for Upstream},
note = {Keynote talk},
keywords = {across, admire, deephealth, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g., clouds, supercomputers, and both of them. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments (such as Kubernetes and SLURM), making it possible to execute onto multiple sites not sharing a common data space. Streamflow clearly distinguishes it from many other workflow management systems because it decouples the data dependencies from the deployment of (containerized) workflow steps. Streamflow also leverages CAPIO (Cross-Application Programmable I/O) to move data from one step to another efficiently. CAPIO captures the POSIX file system and streams it in parallel and in-memory to the workflow's next step, possibly enabling in-transit data filtering.
Marco Aldinucci
From skeletons to workflows in the cloud-edge era Miscellaneous
14th Intl. Symposium on High-Level Programming and Applications (HLPP), 2021, (Keynote talk).
Abstract | Links | BibTeX | Tags: across, admire, deephealth, streamflow
@misc{21:hlpp:streamflow,
title = {From skeletons to workflows in the cloud-edge era},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/RyRPjNBse5PKnab},
year = {2021},
date = {2021-07-01},
address = {Virtual event},
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments and that makes it possible to execute multiple sites not sharing a common data space. StreamFlow supports both task and data parallelism and enables the reproducible and scalable execution of workflows, such as AI pipelines, in hybrid cloud-HPC environments. As a running example, we use the novel ``universal COVID-19 pipeline'' that explore the whole optimisation space of the training of different DNNs to classify COVID-19 lung lesions.},
howpublished = {14th Intl. Symposium on High-Level Programming and Applications (HLPP)},
note = {Keynote talk},
keywords = {across, admire, deephealth, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments and that makes it possible to execute multiple sites not sharing a common data space. StreamFlow supports both task and data parallelism and enables the reproducible and scalable execution of workflows, such as AI pipelines, in hybrid cloud-HPC environments. As a running example, we use the novel ``universal COVID-19 pipeline'' that explore the whole optimisation space of the training of different DNNs to classify COVID-19 lung lesions.
Marco Aldinucci, Iacopo Colonnelli
The Universal Cloud-HPC Pipeline for the AI-Assisted Explainable Diagnosis of COVID-19 Pneumonia Miscellaneous
NVidia GTC'21, 2021, (Invited talk).
Abstract | Links | BibTeX | Tags: deephealth, hpc4ai, streamflow
@misc{21:gtc:clairecovid,
title = {The Universal Cloud-HPC Pipeline for the AI-Assisted Explainable Diagnosis of COVID-19 Pneumonia},
author = {Marco Aldinucci and Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/AkQLbPpEEtDzbbm},
year = {2021},
date = {2021-04-01},
address = {Virtual event},
abstract = {We'll present a methodology to run DNN pipelines on hybrid cloud+HPC infrastructure. We'll also define a "universal pipeline" for medical images. The pipeline can reproduce all state-of-the-art DNNs to diagnose COVID-19 pneumonia, which appeared in the literature during the first Italian lockdown and following months. We can run all of them (across cloud+HPC platforms) and compare their performance in terms of sensitivity and specificity to set a baseline to evaluate future progress in the automated diagnosis of COVID-19. Also, the pipeline makes existing DNNs explainable by way of adversarial training. The pipeline is easily portable and can run across different infrastructures, adapting the performance-urgency trade-off. The methodology builds onto two novel software programs: the streamflow workflow system and the AI-sandbox concept (parallel container with user-space encrypted file system). We reach over 92% accuracy in diagnosing COVID pneumonia.},
howpublished = {NVidia GTC'21},
note = {Invited talk},
keywords = {deephealth, hpc4ai, streamflow},
pubstate = {published},
tppubtype = {misc}
}
We'll present a methodology to run DNN pipelines on hybrid cloud+HPC infrastructure. We'll also define a "universal pipeline" for medical images. The pipeline can reproduce all state-of-the-art DNNs to diagnose COVID-19 pneumonia, which appeared in the literature during the first Italian lockdown and following months. We can run all of them (across cloud+HPC platforms) and compare their performance in terms of sensitivity and specificity to set a baseline to evaluate future progress in the automated diagnosis of COVID-19. Also, the pipeline makes existing DNNs explainable by way of adversarial training. The pipeline is easily portable and can run across different infrastructures, adapting the performance-urgency trade-off. The methodology builds onto two novel software programs: the streamflow workflow system and the AI-sandbox concept (parallel container with user-space encrypted file system). We reach over 92% accuracy in diagnosing COVID pneumonia.
Iacopo Colonnelli
StreamFlow: cross breeding cloud with HPC Miscellaneous
2021 CWL Mini Conference, 2021, (Invited talk).
Abstract | Links | BibTeX | Tags: deephealth, streamflow
@misc{21:CWLMiniConference,
title = {StreamFlow: cross breeding cloud with HPC},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/Le9gg4PfjRxBwXD},
year = {2021},
date = {2021-02-01},
address = {Virtual event},
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space.},
howpublished = {2021 CWL Mini Conference},
note = {Invited talk},
keywords = {deephealth, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space.
2020
Iacopo Colonnelli
StreamFlow: cross breeding cloud with HPC Miscellaneous
HPC-Europa3 2nd Transnational Access Meeting (TAM), 2020, (Invited talk).
Abstract | Links | BibTeX | Tags: streamflow
@misc{20:HPCEuropa3TAM,
title = {StreamFlow: cross breeding cloud with HPC},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/qPHHrSNxk8QXJDw},
year = {2020},
date = {2020-10-01},
address = {Virtual event},
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.},
howpublished = {HPC-Europa3 2nd Transnational Access Meeting (TAM)},
note = {Invited talk},
keywords = {streamflow},
pubstate = {published},
tppubtype = {misc}
}
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.
2019
Iacopo Colonnelli
StreamFlow: un approccio dichiarativo a workflow e pipeline di micro-servizi Miscellaneous
Workshop GARR 2019, 2019.
Abstract | Links | BibTeX | Tags: streamflow
@misc{19:GarrWorkshop,
title = {StreamFlow: un approccio dichiarativo a workflow e pipeline di micro-servizi},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/kZqyiQnBEQNdXJe},
year = {2019},
date = {2019-10-01},
address = {Roma, Italy},
abstract = {Negli ultimi anni, gli approcci orientati ai container si sono dimostrati particolarmente efficaci nel garantire portabilità e riproducibilità dei workflow scientifici. Tuttavia, con il continuo aumento del volume di dati a disposizione e la crescente complessità delle procedure di analisi in ogni campo della ricerca, anche i requisiti di performance e riusabilità si fanno via via sempre più essenziali. L'obiettivo principale di StreamFlow è quello di fornire un nuovo paradigma, totalmente dichiarativo, per la descrizione e l'accelerazione di workflow scientifici in ambienti distribuiti. La peculiarità di StreamFlow risiede nel fatto che l'ambiente di esecuzione è interamente descritto in termini di servizi (container), connessioni tra essi e fattori di replica. Inoltre, ogni task del workflow è esplicitamente mappato sulla tipologia di servizio richiesta. Questo permette un maggior controllo sull'utilizzo delle risorse e politiche di scheduling più precise, a vantaggio delle performance. I principali vantaggi di un approccio dichiarativo sono invece la più facile comprensione ed estensione dei modelli esistenti, a vantaggio della riusabilitià.},
howpublished = {Workshop GARR 2019},
keywords = {streamflow},
pubstate = {published},
tppubtype = {misc}
}
Negli ultimi anni, gli approcci orientati ai container si sono dimostrati particolarmente efficaci nel garantire portabilità e riproducibilità dei workflow scientifici. Tuttavia, con il continuo aumento del volume di dati a disposizione e la crescente complessità delle procedure di analisi in ogni campo della ricerca, anche i requisiti di performance e riusabilità si fanno via via sempre più essenziali. L'obiettivo principale di StreamFlow è quello di fornire un nuovo paradigma, totalmente dichiarativo, per la descrizione e l'accelerazione di workflow scientifici in ambienti distribuiti. La peculiarità di StreamFlow risiede nel fatto che l'ambiente di esecuzione è interamente descritto in termini di servizi (container), connessioni tra essi e fattori di replica. Inoltre, ogni task del workflow è esplicitamente mappato sulla tipologia di servizio richiesta. Questo permette un maggior controllo sull'utilizzo delle risorse e politiche di scheduling più precise, a vantaggio delle performance. I principali vantaggi di un approccio dichiarativo sono invece la più facile comprensione ed estensione dei modelli esistenti, a vantaggio della riusabilitià.