Software & Tools
RISC-V Tools
We actively contribute to the RISC-V ecosystem via different software ports. As of today, these include:
- Fastflow. Fastflow is a C++ framework for high-level pattern-based parallel programming performance. The RISC-V port is part of the official repository.
- Pytorch. PyTorch is one of the most popular Python/C++ frameworks for training and using DNN models. The RISC-V port is available here.
- OpenFL for RISC-V. We managed to port the official Intel® OpenFL federated learning framework to the RISC-V platform. The Python packages are available to be installed via pip from this repository.
To properly add our proprietary repository to your pip configuration, just run:
pip config set global.index-url https://gitlab.di.unito.it/api/v4/projects/1057/packages/pypi/simpleThen, to install OpenFL built for RISC-V, just run:
pip install openfl-riscvAs side results, on this repository are also available the following RISC-V compatible Python packages: ninja (ninja-riscv), meson-python (meson-python-riscv), scipy (scipy-riscv), scikit-learn (scikit-learn-riscv).
2025
2025
Analysis of Model Parallelism for AI Applications on a 64-core RV64 Server CPU
Giulio Malenza, Adriano Marques Garcia, Robert Birke, Luca Benini, Marco Aldinucci
International Journal of Parallel Programming
dyman eupilot icsc riscv
Massive Data Parallel workloads, driven by inference on large ML models, are pushing hardware vendors to develop efficient and cost-effective multi-core server CPUs. The RISC-V architecture plays a prominent role due to its open, extensible, and energy-friendly ISA. Despite significant progress in recent years, finding efficient methods to run AI applications in parallel on new architectures to fully harness their maximum performance remains a challenge. In this study, we investigate the impact of model parallelism on the inference of machine learning models on the SOPHON SG2042 SoC, the first server-grade CPU based on the RV64 ISA, composed of 64 cores arranged in a grid of 16 groups of 4 cores. Specifically, we aim to enhance performance via better data locality stemming from splitting and assigning parts of the model to specific (groups of) cores handling dependencies via a pipeline execution. We orchestrate execution using FastFlow, a low-level programming framework designed for multithreaded streaming applications. By comparing the results against the standard multi-core inference approach based on data parallelism and analyzing the effects of different submodel-to-core mapping strategies, we aim to provide a comprehensive understanding of how the model parallel approach can maximize efficiency and utilization of hardware resources. In our experiments, using model parallelism improved up to 8.4 times the performance over the native PyTorch parallelism.
@article{25:RISCV64:HLPP,
abstract = {Massive Data Parallel workloads, driven by inference on large ML models, are pushing hardware vendors to develop efficient and cost-effective multi-core server CPUs. The RISC-V architecture plays a prominent role due to its open, extensible, and energy-friendly ISA. Despite significant progress in recent years, finding efficient methods to run AI applications in parallel on new architectures to fully harness their maximum performance remains a challenge. In this study, we investigate the impact of model parallelism on the inference of machine learning models on the SOPHON SG2042 SoC, the first server-grade CPU based on the RV64 ISA, composed of 64 cores arranged in a grid of 16 groups of 4 cores. Specifically, we aim to enhance performance via better data locality stemming from splitting and assigning parts of the model to specific (groups of) cores handling dependencies via a pipeline execution. We orchestrate execution using FastFlow, a low-level programming framework designed for multithreaded streaming applications. By comparing the results against the standard multi-core inference approach based on data parallelism and analyzing the effects of different submodel-to-core mapping strategies, we aim to provide a comprehensive understanding of how the model parallel approach can maximize efficiency and utilization of hardware resources. In our experiments, using model parallelism improved up to 8.4 times the performance over the native PyTorch parallelism.},
author = {Malenza, Giulio and Garcia, Adriano Marques and Birke, Robert and Benini, Luca and Aldinucci, Marco},
date = {2025/06/30},
date-added = {2025-07-01 00:02:01 +0200},
date-modified = {2025-07-01 00:02:13 +0200},
doi = {10.1007/s10766-025-00802-6},
isbn = {1573-7640},
journal = {International Journal of Parallel Programming},
keywords = {icsc, eupilot, dyman, riscv},
number = {4},
pages = {27},
title = {Analysis of Model Parallelism for AI Applications on a 64-core RV64 Server CPU},
volume = {53},
year = {2025},
bdsk-url-1 = {https://doi.org/10.1007/s10766-025-00802-6}} A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms
Cristina Silvano, Daniele Ielmini, Fabrizio Ferrandi, Leandro Fiorin, Serena Curzel, Luca Benini, Francesco Conti, Angelo Garofalo, Cristian Zambelli, Enrico Calore, Sebastiano Schifano, Maurizio Palesi, Giuseppe Ascia, Davide Patti, Nicola Petra, Davide De Caro, Luciano Lavagno, Teodoro Urso, Valeria Cardellini, Gian Carlo Cardarilli, Robert Birke, Stefania Perri
ACM Comput. Surv.
icsc riscv
Recent trends in deep learning (DL) have made hardware accelerators essential for various high-performance computing (HPC) applications, including image classification, computer vision, and speech recognition. This survey summarizes and classifies the most recent developments in DL accelerators, focusing on their role in meeting the performance demands of HPC applications. We explore cutting-edge approaches to DL acceleration, covering not only GPU- and TPU-based platforms but also specialized hardware such as FPGA- and ASIC-based accelerators, Neural Processing Units, open hardware RISC-V-based accelerators, and co-processors. This survey also describes accelerators leveraging emerging memory technologies and computing paradigms, including 3D-stacked Processor-In-Memory, non-volatile memories like Resistive RAM and Phase Change Memories used for in-memory computing, as well as Neuromorphic Processing Units, and Multi-Chip Module-based accelerators. Furthermore, we provide insights into emerging quantum-based accelerators and photonics. Finally, this survey categorizes the most influential architectures and technologies from recent years, offering readers a comprehensive perspective on the rapidly evolving field of deep learning acceleration.
@article{10.1145/3729215,
abstract = {Recent trends in deep learning (DL) have made hardware accelerators essential for various high-performance computing (HPC) applications, including image classification, computer vision, and speech recognition. This survey summarizes and classifies the most recent developments in DL accelerators, focusing on their role in meeting the performance demands of HPC applications. We explore cutting-edge approaches to DL acceleration, covering not only GPU- and TPU-based platforms but also specialized hardware such as FPGA- and ASIC-based accelerators, Neural Processing Units, open hardware RISC-V-based accelerators, and co-processors. This survey also describes accelerators leveraging emerging memory technologies and computing paradigms, including 3D-stacked Processor-In-Memory, non-volatile memories like Resistive RAM and Phase Change Memories used for in-memory computing, as well as Neuromorphic Processing Units, and Multi-Chip Module-based accelerators. Furthermore, we provide insights into emerging quantum-based accelerators and photonics. Finally, this survey categorizes the most influential architectures and technologies from recent years, offering readers a comprehensive perspective on the rapidly evolving field of deep learning acceleration.},
address = {New York, NY, USA},
articleno = {286},
author = {Silvano, Cristina and Ielmini, Daniele and Ferrandi, Fabrizio and Fiorin, Leandro and Curzel, Serena and Benini, Luca and Conti, Francesco and Garofalo, Angelo and Zambelli, Cristian and Calore, Enrico and Schifano, Sebastiano and Palesi, Maurizio and Ascia, Giuseppe and Patti, Davide and Petra, Nicola and De Caro, Davide and Lavagno, Luciano and Urso, Teodoro and Cardellini, Valeria and Cardarilli, Gian Carlo and Birke, Robert and Perri, Stefania},
doi = {10.1145/3729215},
issn = {0360-0300},
issue_date = {November 2025},
journal = {ACM Comput. Surv.},
keywords = {icsc, riscv},
month = jun,
number = {11},
numpages = {39},
publisher = {Association for Computing Machinery},
title = {A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms},
url = {https://doi.org/10.1145/3729215},
volume = {57},
year = {2025},
bdsk-url-1 = {https://doi.org/10.1145/3729215}} 2023
2023
Experimenting with PyTorch on RISC-V
Iacopo Colonnelli, Robert Birke, Marco Aldinucci
RISC-V Summit Europe 2023
eupilot icsc riscv
RISC-V is an emerging instruction set architecture. Its modular and extensible open-source royalty-free design is increasingly attracting interest from both research and industry. Nowadays, different RISC-V-based boards can be bought off the shelf. However, software availability is equivalently vital in guaranteeing the RISC-V ecosystem's success. Here we contribute with the first publicly available port of PyTorch. PyTorch is one of the most popular Deep Learning libraries available today. As such, it is a crucial enabler in running state-of-the-art AI applications on RISC-V-based systems and a first step towards a fully democratic end-to-end codesign process.
@inproceedings{23:risc-v-summit,
abstract = {RISC-V is an emerging instruction set architecture. Its modular and extensible open-source royalty-free design is increasingly attracting interest from both research and industry. Nowadays, different RISC-V-based boards can be bought off the shelf. However, software availability is equivalently vital in guaranteeing the RISC-V ecosystem's success. Here we contribute with the first publicly available port of PyTorch. PyTorch is one of the most popular Deep Learning libraries available today. As such, it is a crucial enabler in running state-of-the-art AI applications on RISC-V-based systems and a first step towards a fully democratic end-to-end codesign process.},
address = {Barcelona, Spain},
author = {Iacopo Colonnelli and Robert Birke and Marco Aldinucci},
booktitle = {{RISC-V Summit Europe 2023}},
keywords = {eupilot, icsc, riscv},
month = jun,
note = {Poster},
title = {Experimenting with {PyTorch} on {RISC-V}},
url = {https://iris.unito.it/retrieve/429bf344-9090-42c3-809c-1b8ac320a930/2023-06-08-Iacopo-COLONNELLI-abstract.pdf},
year = {2023},
bdsk-url-1 = {https://iris.unito.it/retrieve/429bf344-9090-42c3-809c-1b8ac320a930/2023-06-08-Iacopo-COLONNELLI-abstract.pdf}} RISC-V-based Platforms for HPC: Analyzing Non-functional Properties for Future HPC and Big-Data Clusters
William Fornaciari, Federico Reghenzani, Federico Terraneo, Davide Baroffio, Cecilia Metra, Martin Omana, Josie E. Rodriguez Condia, Matteo Sonza Reorda, Robert Birke, Iacopo Colonnelli, Gianluca Mittone, Marco Aldinucci, Gabriele Mencagli, Francesco Iannone, Filippo Palombi, Giuseppe Zummo, Daniele Cesarini, Federico Tesser
Embedded Computer Systems: Architectures, Modeling, and Simulation - 23rd International Conference, SAMOS 2023
hpc icsc riscv
High-PerformanceComputing(HPC)haveevolvedtobeused to perform simulations of systems where physical experimentation is pro- hibitively impractical, expensive, or dangerous. This paper provides a general overview and showcases the analysis of non-functional properties in RISC-V-based platforms for HPCs. In particular, our analyses target the evaluation of power and energy control, thermal management, and reliability assessment of promising systems, structures, and technologies devised for current and future generation of HPC machines. The main set of design methodologies and technologies developed within the activ- ities of the Future and HPC & Big Data spoke of the National Centre of HPC, Big Data and Quantum Computing project are described along with the description of the testbed for experimenting two-phase cooling approaches.
@inproceedings{23:SAMOS,
abstract = {High-PerformanceComputing(HPC)haveevolvedtobeused to perform simulations of systems where physical experimentation is pro- hibitively impractical, expensive, or dangerous. This paper provides a general overview and showcases the analysis of non-functional properties in RISC-V-based platforms for HPCs. In particular, our analyses target the evaluation of power and energy control, thermal management, and reliability assessment of promising systems, structures, and technologies devised for current and future generation of HPC machines. The main set of design methodologies and technologies developed within the activ- ities of the Future and HPC & Big Data spoke of the National Centre of HPC, Big Data and Quantum Computing project are described along with the description of the testbed for experimenting two-phase cooling approaches.},
address = {Samos, Greece},
author = {Fornaciari, William and Reghenzani, Federico and Terraneo, Federico and Baroffio, Davide and Metra, Cecilia and Omana, Martin and Rodriguez Condia, Josie E. and Sonza Reorda, Matteo and Birke, Robert and Colonnelli, Iacopo and Mittone, Gianluca and Aldinucci, Marco and Mencagli, Gabriele and Iannone, Francesco and Palombi, Filippo and Zummo, Giuseppe and Cesarini, Daniele and Tesser, Federico},
booktitle = {{Embedded Computer Systems: Architectures, Modeling, and Simulation - 23rd International Conference, {SAMOS} 2023}},
doi = {10.1007/978-3-031-46077-7_26},
keywords = {icsc, riscv, hpc},
note = {icsc},
title = {{RISC-V}-based Platforms for {HPC}: Analyzing Non-functional Properties for Future {HPC} and {Big-Data} Clusters},
url = {https://iris.unito.it/retrieve/b627eab0-3aa1-4fd7-8685-f47c62c792b3/SAMOS_2023_CN_HPC_FL1.pdf},
year = {2023},
bdsk-url-1 = {https://iris.unito.it/retrieve/b627eab0-3aa1-4fd7-8685-f47c62c792b3/SAMOS_2023_CN_HPC_FL1.pdf}} Model-Agnostic Federated Learning
Gianluca Mittone, Walter Riviera, Iacopo Colonnelli, Robert Birke, Marco Aldinucci
Euro-Par 2023: Parallel Processing
eupilot fl icsc riscv
Since its debut in 2016, Federated Learning (FL) has been tied to the inner workings of Deep Neural Networks (DNNs). On the one hand, this allowed its development and widespread use as DNNs proliferated. On the other hand, it neglected all those scenarios in which using DNNs is not possible or advantageous. The fact that most current FL frameworks only allow training DNNs reinforces this problem. To address the lack of FL solutions for non-DNN-based use cases, we propose MAFL (Model-Agnostic Federated Learning). MAFL marries a model-agnostic FL algorithm, AdaBoost.F, with an open industry-grade FL framework: Intel OpenFL. MAFL is the first FL system not tied to any specific type of machine learning model, allowing exploration of FL scenarios beyond DNNs and trees. We test MAFL from multiple points of view, assessing its correctness, flexibility and scaling properties up to 64 nodes. We optimised the base software achieving a 5.5x speedup on a standard FL scenario. MAFL is compatible with x86-64, ARM-v8, Power and RISC-V.
@inproceedings{23:mittone:mafl,
abstract = {Since its debut in 2016, Federated Learning (FL) has been tied to the inner workings of Deep Neural Networks (DNNs). On the one hand, this allowed its development and widespread use as DNNs proliferated. On the other hand, it neglected all those scenarios in which using DNNs is not possible or advantageous. The fact that most current FL frameworks only allow training DNNs reinforces this problem. To address the lack of FL solutions for non-DNN-based use cases, we propose MAFL (Model-Agnostic Federated Learning). MAFL marries a model-agnostic FL algorithm, AdaBoost.F, with an open industry-grade FL framework: Intel OpenFL. MAFL is the first FL system not tied to any specific type of machine learning model, allowing exploration of FL scenarios beyond DNNs and trees. We test MAFL from multiple points of view, assessing its correctness, flexibility and scaling properties up to 64 nodes. We optimised the base software achieving a 5.5x speedup on a standard FL scenario. MAFL is compatible with x86-64, ARM-v8, Power and RISC-V.},
address = {Limassol, Cyprus},
author = {Mittone, Gianluca and Riviera, Walter and Colonnelli, Iacopo and Birke, Robert and Aldinucci, Marco},
booktitle = {Euro-Par 2023: Parallel Processing},
date-added = {2023-03-8 21:51:14 +0000},
date-modified = {2024-08-18 13:15:50 +0200},
doi = {10.1007/978-3-031-39698-4_26},
institution = {Computer Science Department, University of Torino},
keywords = {eupilot, icsc, riscv, fl},
month = aug,
pages = {383--396},
publisher = {{Springer}},
title = {Model-Agnostic Federated Learning},
url = {https://hdl.handle.net/2318/1928750},
volume = {14100},
year = {2023},
bdsk-url-1 = {https://doi.org/10.48550/arXiv.2303.04906}} Experimenting with Emerging RISC-V Systems for Decentralised Machine Learning
Gianluca Mittone, Nicolò Tonci, Robert Birke, Iacopo Colonnelli, Doriana Medić, Andrea Bartolini, Roberto Esposito, Emanuele Parisi, Francesco Beneventi, Mirko Polato, Massimo Torquati, Luca Benini, Marco Aldinucci
20th ACM International Conference on Computing Frontiers (CF '23)
eupilot fl hpc icsc riscv
Decentralised Machine Learning (DML) enables collaborative machine learning without centralised input data. Federated Learning (FL) and Edge Inference are examples of DML. While tools for DML (especially FL) are starting to flourish, many are not flexible and portable enough to experiment with novel systems (e.g., RISC-V), non-fully connected topologies, and asynchronous collaboration schemes. We overcome these limitations via a domain-specific language allowing to map DML schemes to an underlying middleware, i.e. the FastFlow parallel programming library. We experiment with it by generating different working DML schemes on two emerging architectures (ARM-v8, RISC-V) and the x86-64 platform. We characterise the performance and energy efficiency of the presented schemes and systems. As a byproduct, we introduce a RISC-V porting of the PyTorch framework, the first publicly available to our knowledge.
@inproceedings{23:mittone:fl-riscv,
abstract = {Decentralised Machine Learning (DML) enables collaborative machine learning without centralised input data. Federated Learning (FL) and Edge Inference are examples of DML. While tools for DML (especially FL) are starting to flourish, many are not flexible and portable enough to experiment with novel systems (e.g., RISC-V), non-fully connected topologies, and asynchronous collaboration schemes. We overcome these limitations via a domain-specific language allowing to map DML schemes to an underlying middleware, i.e. the FastFlow parallel programming library. We experiment with it by generating different working DML schemes on two emerging architectures (ARM-v8, RISC-V) and the x86-64 platform. We characterise the performance and energy efficiency of the presented schemes and systems. As a byproduct, we introduce a RISC-V porting of the PyTorch framework, the first publicly available to our knowledge.},
address = {Bologna, Italy},
author = {Mittone, Gianluca and Tonci, Nicol{\`o} and Birke, Robert and Colonnelli, Iacopo and Medi\'{c}, Doriana and Bartolini, Andrea and Esposito, Roberto and Parisi, Emanuele and Beneventi, Francesco and Polato, Mirko and Torquati, Massimo and Benini, Luca and Aldinucci, Marco},
booktitle = {20th {ACM} International Conference on Computing Frontiers ({CF} '23)},
date-added = {2023-03-14 15:34:00 +0000},
date-modified = {2024-08-18 13:15:37 +0200},
doi = {10.1145/3587135.3592211},
institution = {Computer Science Department, University of Torino},
isbn = {979-8-4007-0140-5/23/05},
keywords = {eupilot, icsc, hpc, riscv, fl},
month = may,
note = {https://arxiv.org/abs/2302.07946},
publisher = {{ACM}},
title = {Experimenting with Emerging {RISC-V} Systems for Decentralised Machine Learning},
url = {https://dl.acm.org/doi/pdf/10.1145/3587135.3592211},
year = {2023},
bdsk-url-1 = {https://dl.acm.org/doi/pdf/10.1145/3587135.3592211},
bdsk-url-2 = {https://doi.org/10.1145/3587135.3592211}}Cross-Facility Federated Learning (xFFL)
Cross-Facility Federated Learning (xFFL) is a federated learning (FL) framework based on the StreamFlow workflow management system (WMS) developed in the Parallel Computing [Alpha] research group at the University of Turin, Italy.
xFFL is designed as research-oriented software, enabling FL workload deployment on geographically distributed computing infrastructures, including, but not limited to, high-performance computers (HPCs).
We aim to democratize extensive ML workload development, allowing researchers and SMEs to be competitive with big tech companies by exploiting sparse computing power.
FastFederatedLearning
Fast Federated Learning (FFL) is a C/C++-based Federated Learning framework built on top of the parallel programming FastFlow framework. It exploits the Cereal library to efficiently serialise the updates sent over the network and the libtorch library to fully bypass the need for Python code. The first release of this software comprises three examples based on three different communication topologies: master-worker, peer-to-peer, and tree-based.
FastFederatedLearning is freely available on GitHub under the LGPLv3 license. It has been successfully tested on x86_64, ARM, and RISC-V platforms. FFL has scripts for automatically installing the framework and reproducing all the experiments reported in the original paper. More information about software usage can be found on the official repository.
Model-Agnostic Federated Learning (MAFL)
Model-Agnostic Federated Learning (MAFL, also known as OpenFL-extended) is an extended version of the official Intel® OpenFL federated learning (FL) framework. MAFL fully supports the standard FL workflow already provided by OpenFL, but in addition, it provides support for both federated bagging and federated boosting approaches. Federated bagging is implemented through simple bagging of models trained by different parties from the aggregator, while federated boosting is obtained employing the AdaBoost.F algorithm developed at the University of Turin[1]. Through these approaches, MAFL is fully model-agnostic, which means that it can be used to build federations out of any Machine Learning model, not only Deep Neural Networks.
MAFL is freely available on GitHub under the LGPLv3 license. It is fully Python-based and comes with a wide range of ready-made examples. It has been tested on x86_64, ARM and RISC-V architectures. More information about software usage can be found on the official repository.
[1] M. Polato, R. Esposito, and M. Aldinucci. “Boosting the federation: Cross-silo federated learning without gradient descent.” 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022.
Jupyter Workflow
Jupyter Workflow is an extension of the IPython kernel designed to support distributed literate workflows. The Jupyter Workflow kernel enables Jupyter Notebooks to describe complex workflows and to execute them in a distributed fashion on hybrid cloud/HPC infrastructures. In particular, code cells are regarded as the nodes of a distributed workflow graph, whereas cell metadata are used to express data and control dependencies, parallel execution patterns (e.g. Scatter/Gather), and target execution infrastructures.
Jupyter Workflow code is available on GitHub under the LGPLv3 license, and the related Python package is downloadable from PyPI. More details about the tool and its applications can be found on the Jupyter Workflow website.
I. Colonnelli, M. Aldinucci, B. Cantalupo, L. Padovani, S. Rabellino, C. Spampinato, R. Morelli, R. Di Carlo, N. Magini and C. Cavazzoni, “Distributed workflows with Jupyter”, Future Generation Computer Systems, vol. 128, pp. 282-298, 2022. doi: 10.1016/j.future.2021.10.007.
2026
2026
The Italian pathfinding strategy for RISC-V-based HPC: Progress within DARE-SGA1 project
Giovanni Agosta, Marco Aldinucci, Marcello Barbirotta, Andrea Bartolini, Laura Bellentani, Andrea Biagioni, Daniele Cattaneo, Daniele Cesarini, Carlotta Chiarini, Iacopo Colonnelli, Francesco Conti, Pietro Delugas, Lev Denisov, Ottorino Frezza, Angelo Garofalo, Marco Grangetto, Juan David Guerrero Balaguera, Francesca Lo Cicero, Alessandro Lonardo, Michele Martinelli, Andrea Maslov, Francesco Menichelli, Mauro Olivieri, Pierpaolo Perticaroli, Luca Pontisso, Cristian Rossi, Josie Esteban Rodriguez Condia, Daniele Rossi, Davide Rossi, Sergio Saponara, Antonio Sciarappa, Francesco Simula, Matteo Sonza Reorda, Gabriele Spadaro, Massimo Torquati, Piero Vicini
Microprocessors and Microsystems
capio dare jupyter-workflow streamflow
Driven by the need of technological sovereignty, the European Union has accelerated the development of a sustainable High-Performance Computing (HPC) ecosystem. In this perspective, the DARE SGA1 (Digital Autonomy with RISC-V in Europe) project seeks to decouple European infrastructure from proprietary, non-European hardware by developing high-performance processors and computing accelerators based on the open-source RISC-V architecture. This paper introduces and reports about the current status of the multifaceted contributions of Italian partners to the critical pathfinding activities within the DARE SGA1 framework aiming at closing the gap between current prototypes and future HPC systems as well as establishing a bridge towards a fully independent European supercomputing supply chain. In particular, the activities focus on (i) novel RISC-V-based accelerator designs optimized for energy efficiency and high throughput and scaling-out capability; (ii) improvement of RISC-V-based system reliability and resiliency; (iii) evolution of the software stack for low level software and compiler optimizations (iv) co-design through benchmarking with AI and Quantum Chemistry scalable applications.
@article{2026:micpro:dare,
abstract = {Driven by the need of technological sovereignty, the European Union has accelerated the development of a sustainable High-Performance Computing (HPC) ecosystem. In this perspective, the DARE SGA1 (Digital Autonomy with RISC-V in Europe) project seeks to decouple European infrastructure from proprietary, non-European hardware by developing high-performance processors and computing accelerators based on the open-source RISC-V architecture. This paper introduces and reports about the current status of the multifaceted contributions of Italian partners to the critical pathfinding activities within the DARE SGA1 framework aiming at closing the gap between current prototypes and future HPC systems as well as establishing a bridge towards a fully independent European supercomputing supply chain. In particular, the activities focus on (i) novel RISC-V-based accelerator designs optimized for energy efficiency and high throughput and scaling-out capability; (ii) improvement of RISC-V-based system reliability and resiliency; (iii) evolution of the software stack for low level software and compiler optimizations (iv) co-design through benchmarking with AI and Quantum Chemistry scalable applications.},
author = {Giovanni Agosta and Marco Aldinucci and Marcello Barbirotta and Andrea Bartolini and Laura Bellentani and Andrea Biagioni and Daniele Cattaneo and Daniele Cesarini and Carlotta Chiarini and Iacopo Colonnelli and Francesco Conti and Pietro Delugas and Lev Denisov and Ottorino Frezza and Angelo Garofalo and Marco Grangetto and Juan David Guerrero Balaguera and Francesca {Lo Cicero} and Alessandro Lonardo and Michele Martinelli and Andrea Maslov and Francesco Menichelli and Mauro Olivieri and Pierpaolo Perticaroli and Luca Pontisso and Cristian Rossi and Josie Esteban Rodriguez Condia and Daniele Rossi and Davide Rossi and Sergio Saponara and Antonio Sciarappa and Francesco Simula and Matteo Sonza Reorda and Gabriele Spadaro and Massimo Torquati and Piero Vicini},
doi = {10.1016/j.micpro.2026.105321},
issn = {0141-9331},
journal = {Microprocessors and Microsystems},
keywords = {dare, streamflow, jupyter-workflow, capio},
pages = {105321},
title = {The Italian pathfinding strategy for {RISC-V}-based {HPC}: Progress within {DARE-SGA1} project},
url = {https://iris.unito.it/retrieve/e6b5b8a2-a130-4123-9f3b-22f2befeadd0/DARE_ICSC_IEEE_MICPRO.pdf},
volume = {124},
year = 2026,
bdsk-url-1 = {https://iris.unito.it/retrieve/e6b5b8a2-a130-4123-9f3b-22f2befeadd0/DARE_ICSC_IEEE_MICPRO.pdf},
bdsk-url-2 = {https://doi.org/10.1016/j.micpro.2026.105321}} 2025
2025
The Cloud-HPC infrastructure for Hazard Mapping and vulnerability Monitoring (HaMMon)
Mauro Imbrosciano, Eva Sciacca, Fabio Vitello, Leonardo Pelonero, Francesco Franchina, Ugo Becciani, Iacopo Colonnelli, Doriana Medić
2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP)
hpc4ai icsc jupyter-workflow streamflow system-sw
The HaMMon project is the outcome of an industrial partnership that includes many Italian research institutions and private companies. It is led by UnipolSai and Leitha, and funded by the ICSC, the Italian National Research Center for High Performance Computing, Big Data and Quantum Computing.The ambition of HaMMon is to build a flexible and scalable platform to analyze the hydrogeological and atmospheric balance of the Italian territory. The project aims to expand the current knowledge in hazard mapping, monitoring, and forecasting from an industrial perspective by leveraging innovative technologies and the interdisciplinary activities carried out by the ICSC.In this work, we present the cloud-HPC infrastructure deployed in the High-Performance Computing for Artificial Intelligence (HPC4AI) green data center of the University of Turin which supports the testing and development of HaMMon's applications and services. We describe the current activities and preliminary results related to the integration of Photogrammetry techniques, Data Visualization and Artificial Intelligence technologies, applied on aerial images, to assess extreme natural events and evaluate their impact on risk-exposed assets.
@inproceedings{imbrosciano:2025:pdp,
abstract = {The HaMMon project is the outcome of an industrial partnership that includes many Italian research institutions and private companies. It is led by UnipolSai and Leitha, and funded by the ICSC, the Italian National Research Center for High Performance Computing, Big Data and Quantum Computing.The ambition of HaMMon is to build a flexible and scalable platform to analyze the hydrogeological and atmospheric balance of the Italian territory. The project aims to expand the current knowledge in hazard mapping, monitoring, and forecasting from an industrial perspective by leveraging innovative technologies and the interdisciplinary activities carried out by the ICSC.In this work, we present the cloud-HPC infrastructure deployed in the High-Performance Computing for Artificial Intelligence (HPC4AI) green data center of the University of Turin which supports the testing and development of HaMMon's applications and services. We describe the current activities and preliminary results related to the integration of Photogrammetry techniques, Data Visualization and Artificial Intelligence technologies, applied on aerial images, to assess extreme natural events and evaluate their impact on risk-exposed assets.},
author = {Mauro Imbrosciano and Eva Sciacca and Fabio Vitello and Leonardo Pelonero and Francesco Franchina and Ugo Becciani and Iacopo Colonnelli and Doriana Medi\'{c}},
booktitle = {2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP)},
date-modified = {2025-05-24 09:13:03 +0200},
doi = {10.1109/PDP66500.2025.00050},
issn = {2377-5750},
keywords = {system-sw, hpc4ai, icsc, jupyter-workflow, streamflow},
pages = {309--316},
publisher = {IEEE},
title = {The Cloud-HPC infrastructure for Hazard Mapping and vulnerability Monitoring (HaMMon)},
url = {https://iris.unito.it/retrieve/36a223d5-7134-4469-80a0-705c0060a5d4/PDP_2025_HaMMON.pdf},
year = {2025},
bdsk-url-1 = {https://iris.unito.it/retrieve/36a223d5-7134-4469-80a0-705c0060a5d4/PDP_2025_HaMMON.pdf},
bdsk-url-2 = {https://doi.org/10.1109/PDP66500.2025.00050}} 2023
2023
Workflow Models for Heterogeneous Distributed Systems
Iacopo Colonnelli
Proceedings of the 2nd Italian Conference on Big Data and Data Science (ITADATA 2023), Naples, Italy, September 11-13, 2023
across eupex icsc jupyter-workflow streamflow
This article introduces a novel hybrid workflow abstraction that injects topology awareness directly into the definition of a distributed workflow model. In particular, the article briefly discusses the advantages brought by this approach to the design and orchestration of large-scale data-oriented workflows, the current level of support from state-of-the-art workflow systems, and some future research directions.
@inproceedings{23:colonnelli:itadata,
abstract = {This article introduces a novel hybrid workflow abstraction that injects topology awareness directly into the definition of a distributed workflow model. In particular, the article briefly discusses the advantages brought by this approach to the design and orchestration of large-scale data-oriented workflows, the current level of support from state-of-the-art workflow systems, and some future research directions.},
author = {Iacopo Colonnelli},
booktitle = {Proceedings of the 2nd Italian Conference on Big Data and Data Science ({ITADATA} 2023), Naples, Italy, September 11-13, 2023},
editor = {Bena, Nicola and Di Martino, Beniamino and Maratea, Antonio and Sperduti, Alessandro and Di Nardo, Emanuel and Ciaramella, Angelo and Montella, Raffaele and Ardagna, Claudio A.},
keywords = {across, eupex, icsc, streamflow, jupyter-workflow},
publisher = {CEUR-WS.org},
series = {{CEUR} Workshop Proceedings},
title = {Workflow Models for Heterogeneous Distributed Systems},
url = {https://ceur-ws.org/Vol-3606/invited77.pdf},
volume = {3606},
year = {2023},
bdsk-url-1 = {https://ceur-ws.org/Vol-3606/invited77.pdf}} A Systematic Mapping Study of Italian Research on Workflows
Marco Aldinucci, Elena Maria Baralis, Valeria Cardellini, Iacopo Colonnelli, Marco Danelutto, Sergio Decherchi, Giuseppe Di Modica, Luca Ferrucci, Marco Gribaudo, Francesco Iannone, Marco Lapegna, Doriana Medić, Giuseppa Muscianisi, Francesca Righetti, Eva Sciacca, Nicola Tonellotto, Mauro Tortonesi, Paolo Trunfio, Tullio Vardanega
Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023
icsc jupyter-workflow streamflow
An entire ecosystem of methodologies and tools revolves around scientific workflow management. They cover crucial non-functional requirements that standard workflow models fail to target, such as interactive execution, energy efficiency, performance portability, Big Data management, and intelligent orchestration in the Computing Continuum. Characterizing and monitoring this ecosystem is crucial to develop an informed view of current and future research directions. This work conducts a systematic mapping study of the Italian workflow research community, collecting and analyzing 25 tools and 10 applications from several scientific domains in the context of the “National Research Centre for HPC, Big Data, and Quantum Computing” (ICSC). The study aims to outline the main current research directions and determine how they address the critical needs of modern scientific applications. The findings highlight a variegated research ecosystem of tools, with a prominent interest in advanced workflow orchestration and still immature but promising efforts toward energy efficiency.
@inproceedings{WORKS2023,
abstract = {An entire ecosystem of methodologies and tools revolves around scientific workflow management. They cover crucial non-functional requirements that standard workflow models fail to target, such as interactive execution, energy efficiency, performance portability, Big Data management, and intelligent orchestration in the Computing Continuum. Characterizing and monitoring this ecosystem is crucial to develop an informed view of current and future research directions. This work conducts a systematic mapping study of the Italian workflow research community, collecting and analyzing 25 tools and 10 applications from several scientific domains in the context of the ``National Research Centre for HPC, Big Data, and Quantum Computing'' (ICSC). The study aims to outline the main current research directions and determine how they address the critical needs of modern scientific applications. The findings highlight a variegated research ecosystem of tools, with a prominent interest in advanced workflow orchestration and still immature but promising efforts toward energy efficiency.},
address = {Denver, CO, USA},
author = {Aldinucci, Marco and Maria Baralis, Elena and Cardellini, Valeria and Colonnelli, Iacopo and Danelutto, Marco and Decherchi, Sergio and Di Modica, Giuseppe and Ferrucci, Luca and Gribaudo, Marco and Iannone, Francesco and Lapegna, Marco and Medi\'{c}, Doriana and Muscianisi, Giuseppa and Righetti, Francesca and Sciacca, Eva and Tonellotto, Nicola and Tortonesi, Mauro and Trunfio, Paolo and Vardanega, Tullio},
booktitle = {Proceedings of the {SC} '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, {SC-W} 2023},
doi = {10.1145/3624062.3624285},
keywords = {streamflow, jupyter-workflow, icsc},
month = nov,
pages = {2065--2076},
publisher = {{ACM}},
title = {A Systematic Mapping Study of Italian Research on Workflows},
url = {https://doi.org/10.1145/3624062.3624285},
year = {2023},
bdsk-url-1 = {https://doi.org/10.1145/3624062.3624285}} 2022
2022
Distributed workflows with Jupyter
Iacopo Colonnelli, Marco Aldinucci, Barbara Cantalupo, Luca Padovani, Sergio Rabellino, Concetto Spampinato, Roberto Morelli, Rosario Di Carlo, Nicol`o Magini, Carlo Cavazzoni
Future Generation Computer Systems
across deephealth jupyter-workflow streamflow
The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code's business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks' potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.
@article{21:FGCS:jupyflow,
abstract = {The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code's business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks' potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.},
author = {Colonnelli, Iacopo and Aldinucci, Marco and Cantalupo, Barbara and Padovani, Luca and Rabellino, Sergio and Spampinato, Concetto and Morelli, Roberto and Di Carlo, Rosario and Magini, Nicol{`o} and Cavazzoni, Carlo},
doi = {10.1016/j.future.2021.10.007},
issn = {0167-739X},
journal = {Future Generation Computer Systems},
keywords = {streamflow, jupyter-workflow, deephealth, across},
pages = {282--298},
title = {Distributed workflows with {Jupyter}},
url = {https://www.sciencedirect.com/science/article/pii/S0167739X21003976},
volume = {128},
year = {2022},
bdsk-url-1 = {https://www.sciencedirect.com/science/article/pii/S0167739X21003976},
bdsk-url-2 = {https://doi.org/10.1016/j.future.2021.10.007}} 2024
2024
Dynamic hybrid workflows for Deep Learning on HPC infrastructure
Iacopo Colonnelli
icsc jupyter-workflow streamflow
Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher's laptop to an entire HPC facility.
@misc{24:icolonne:ictp,
abstract = {Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher's laptop to an entire HPC facility.},
address = {Trieste, Italy},
author = {Iacopo Colonnelli},
keywords = {icsc, streamflow, jupyter-workflow},
month = may,
title = {Dynamic hybrid workflows for Deep Learning on HPC infrastructure},
url = {https://datacloud.di.unito.it/index.php/s/EaFHJEKNbW5oXeq},
year = 2024,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/EaFHJEKNbW5oXeq}} 2023
2023
UNITO tools presentation
Iacopo Colonnelli
jupyter-workflow streamflow
@misc{23:FL3WorkingDay,
address = {Bologna, Italy},
author = {Iacopo Colonnelli},
howpublished = {CN HPC Flagship 3 Working Day},
keywords = {streamflow, jupyter-workflow},
month = may,
title = {{UNITO} tools presentation},
url = {https://datacloud.di.unito.it/index.php/s/fgHbnLDQSFtcwLd},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/fgHbnLDQSFtcwLd}} Workflow models for heterogeneous distributed systems
Iacopo Colonnelli
jupyter-workflow streamflow
@misc{23:ITADATABestPhDThesis,
address = {Napoli, Italy},
author = {Iacopo Colonnelli},
howpublished = {2nd Italian Conference on Big Data and Data Science (ITADATA 2023)},
keywords = {streamflow, jupyter-workflow},
month = sep,
note = {Best PhD Thesis Award},
title = {Workflow models for heterogeneous distributed systems},
url = {https://datacloud.di.unito.it/index.php/s/6RqcaJ4djqFNDC8},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/6RqcaJ4djqFNDC8}} 2022
2022
Hybrid Workflows For Large-Scale Scientific Applications
Iacopo Colonnelli, Marco Aldinucci
across eupex jupyter-workflow textarossa
Large-scale scientific applications are facing an irreversible transition from monolithic, high-performance oriented codes to modular and polyglot deployments of specialised (micro-)services. The reasons behind this transition are many: coupling of standard solvers with Deep Learning techniques, offloading of data analysis and visualisation to Cloud, and the advent of specialised hardware accelerators. Topology-aware Workflow Management Systems (WMSs) play a crucial role. In particular, topology-awareness allows an explicit mapping of workflow steps onto heterogeneous locations, allowing automated executions on top of hybrid architectures (e.g., cloud+HPC or classical+quantum). Plus, topology-aware WMSs can offer non-functional requirements OOTB, e.g. components' life-cycle orchestration, secure and efficient data transfers, fault tolerance, and cross-cluster execution of urgent workloads. Augmenting interactive Jupyter Notebooks with distributed workflow capabilities allows domain experts to prototype and scale applications using the same technological stack, while relying on a feature-rich and user-friendly web interface. This abstract will showcase how these general methodologies can be applied to a typical geoscience simulation pipeline based on the Full Wavefront Inversion (FWI) technique. In particular, a prototypical Jupyter Notebook will be executed interactively on Cloud. Preliminary data analyses and post-processing will be executed locally, while the computationally demanding optimisation loop will be scheduled on a remote HPC cluster.
@misc{22:eage,
abstract = {Large-scale scientific applications are facing an irreversible transition from monolithic, high-performance oriented codes to modular and polyglot deployments of specialised (micro-)services. The reasons behind this transition are many: coupling of standard solvers with Deep Learning techniques, offloading of data analysis and visualisation to Cloud, and the advent of specialised hardware accelerators. Topology-aware Workflow Management Systems (WMSs) play a crucial role. In particular, topology-awareness allows an explicit mapping of workflow steps onto heterogeneous locations, allowing automated executions on top of hybrid architectures (e.g., cloud+HPC or classical+quantum). Plus, topology-aware WMSs can offer non-functional requirements OOTB, e.g. components' life-cycle orchestration, secure and efficient data transfers, fault tolerance, and cross-cluster execution of urgent workloads. Augmenting interactive Jupyter Notebooks with distributed workflow capabilities allows domain experts to prototype and scale applications using the same technological stack, while relying on a feature-rich and user-friendly web interface. This abstract will showcase how these general methodologies can be applied to a typical geoscience simulation pipeline based on the Full Wavefront Inversion (FWI) technique. In particular, a prototypical Jupyter Notebook will be executed interactively on Cloud. Preliminary data analyses and post-processing will be executed locally, while the computationally demanding optimisation loop will be scheduled on a remote HPC cluster.},
address = {Milano, Italy},
author = {Iacopo Colonnelli and Marco Aldinucci},
howpublished = {6th EAGE High Performance Computing Workshop},
keywords = {eupex, across, textarossa, jupyter-workflow},
month = sep,
title = {Hybrid Workflows For Large-Scale Scientific Applications},
url = {https://datacloud.di.unito.it/index.php/s/GScPS5LCPdt6Yoo},
year = {2022},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/GScPS5LCPdt6Yoo}} Dossier: multi-tenant distributed Jupyter Notebooks
Iacopo Colonnelli, Dario Tranchitella
across deephealth hpc4ai jupyter-workflow
When providing data analysis as a service, one must tackle several problems. Data privacy and protection by design are crucial when working on sensitive data. Performance and scalability are fundamental for compute-intensive workloads, e.g. training Deep Neural Networks. User-friendly interfaces and fast prototyping tools are essential to allow domain experts to experiment with new techniques. Portability and reproducibility are necessary to assess the actual value of results. Kubernetes is the best platform to provide reliable, elastic, and maintainable services. However, Kubernetes alone is not enough to achieve large-scale multi-tenant reproducible data analysis. OOTB support for multi-tenancy is too rough, with only two levels of segregation (i.e. the single namespace or the entire cluster). Offloading computation to off-cluster resources is non-trivial and requires the user's manual configuration. Also, Jupyter Notebooks per se cannot provide much scalability (they execute locally and sequentially) and reproducibility (users can run cells in any order and any number of times). The Dossier platform allows system administrators to manage multi-tenant distributed Jupyter Notebooks at the cluster level in the Kubernetes way, i.e. through CRDs. Namespaces are aggregated in Tenants, and all security and accountability aspects are managed at that level. Each Notebook spawns into a user-dedicated namespace, subject to all Tenant-level constraints. Users can rely on provisioned resources, either in-cluster worker nodes or external resources like HPC facilities. Plus, they can plug their computing nodes in a BYOD fashion. Notebooks are interpreted as distributed workflows, where each cell is a task that one can offload to a different location in charge of its execution.
@misc{22:data-on-kubernetes,
abstract = {When providing data analysis as a service, one must tackle several problems. Data privacy and protection by design are crucial when working on sensitive data. Performance and scalability are fundamental for compute-intensive workloads, e.g. training Deep Neural Networks. User-friendly interfaces and fast prototyping tools are essential to allow domain experts to experiment with new techniques. Portability and reproducibility are necessary to assess the actual value of results. Kubernetes is the best platform to provide reliable, elastic, and maintainable services. However, Kubernetes alone is not enough to achieve large-scale multi-tenant reproducible data analysis. OOTB support for multi-tenancy is too rough, with only two levels of segregation (i.e. the single namespace or the entire cluster). Offloading computation to off-cluster resources is non-trivial and requires the user's manual configuration. Also, Jupyter Notebooks per se cannot provide much scalability (they execute locally and sequentially) and reproducibility (users can run cells in any order and any number of times). The Dossier platform allows system administrators to manage multi-tenant distributed Jupyter Notebooks at the cluster level in the Kubernetes way, i.e. through CRDs. Namespaces are aggregated in Tenants, and all security and accountability aspects are managed at that level. Each Notebook spawns into a user-dedicated namespace, subject to all Tenant-level constraints. Users can rely on provisioned resources, either in-cluster worker nodes or external resources like HPC facilities. Plus, they can plug their computing nodes in a BYOD fashion. Notebooks are interpreted as distributed workflows, where each cell is a task that one can offload to a different location in charge of its execution.},
address = {Virtual event},
author = {Iacopo Colonnelli and Dario Tranchitella},
howpublished = {DoK Talks 141},
keywords = {jupyter-workflow, across, deephealth, hpc4ai},
month = jul,
note = {Invited talk},
title = {Dossier: multi-tenant distributed {J}upyter {N}otebooks},
url = {https://datacloud.di.unito.it/index.php/s/RNqTGmTqWS66qHT},
year = {2022},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/RNqTGmTqWS66qHT}} OpenDeepHealth: Crafting a Deep Learning Platform as a Service with Kubernetes
Iacopo Colonnelli, Dario Tranchitella
across deephealth hpc4ai jupyter-workflow streamflow
@misc{22:jotb22,
address = {Malaga, Spain},
author = {Iacopo Colonnelli and Dario Tranchitella},
howpublished = {J on The Beach 2022},
keywords = {streamflow, jupyter-workflow, across, deephealth, hpc4ai},
month = apr,
title = {{OpenDeepHealth}: Crafting a Deep Learning Platform as a Service with {K}ubernetes},
url = {https://datacloud.di.unito.it/index.php/s/n6J7STNnwdyqtET},
year = {2022},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/n6J7STNnwdyqtET}} Distributed workflows with Jupyter
Iacopo Colonnelli
across deephealth jupyter-workflow streamflow
@misc{22:jotb22-workshop,
address = {Malaga, Spain},
author = {Iacopo Colonnelli},
howpublished = {J on The Beach 2022},
keywords = {streamflow, jupyter-workflow, deephealth, across},
month = apr,
note = {Workshop},
title = {Distributed workflows with {J}upyter},
url = {https://datacloud.di.unito.it/index.php/s/om89q55S6ePf2Ji},
year = {2022},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/om89q55S6ePf2Ji}} 2020
2020
JupyterFlow: Jupyter Notebooks su larga scala
Iacopo Colonnelli, Sergio Rabellino
deephealth hpc4ai jupyter-workflow
I Jupyter Notebook sono largamente utilizzati sia in ambito industriale che accademico come strumento di didattica, prototipazione e analisi esplorative. Purtroppo il sistema runtime standard di Jupyter non è abbastanza potente per sostenere un carichi di lavoro reali e spesso l'unica soluzione è quella di riscrivere il codice da zero in una tecnologia con supporto HPC. Intrgrando lo stack Jupyter con StreamFlow (https://streamflow.di.unito.it/) è possibile creare i Notebook tramite un'interfaccia web su cloud ed eseguirli in maniera trasparente in remoto su una VM con GPU o su nodi HPC.
@misc{20:GarrWorkshop,
abstract = {I Jupyter Notebook sono largamente utilizzati sia in ambito industriale che accademico come strumento di didattica, prototipazione e analisi esplorative. Purtroppo il sistema runtime standard di Jupyter non \`{e} abbastanza potente per sostenere un carichi di lavoro reali e spesso l'unica soluzione \`{e} quella di riscrivere il codice da zero in una tecnologia con supporto HPC. Intrgrando lo stack Jupyter con StreamFlow (https://streamflow.di.unito.it/) \`{e} possibile creare i Notebook tramite un'interfaccia web su cloud ed eseguirli in maniera trasparente in remoto su una VM con GPU o su nodi HPC.},
address = {Virtual event},
author = {Iacopo Colonnelli and Sergio Rabellino},
howpublished = {Workshop GARR 2020},
keywords = {jupyter-workflow, hpc4ai, deephealth},
month = nov,
title = {{JupyterFlow}: {J}upyter {N}otebooks su larga scala},
url = {https://datacloud.di.unito.it/index.php/s/ASPEmyXAj5QscgC},
year = {2020},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/ASPEmyXAj5QscgC},
bdsk-url-2 = {https://www.eventi.garr.it/it/ws20/programma/speaker/680-iacopo-colonnelli}}
StreamFlow
The StreamFlow framework is a container-native Workflow Management System (WMS) written in Python 3 and based on the Common Workflow Language (CWL) Standard.
StreamFlow has been designed around two main principles:
- Allowing the execution of tasks in multi-container environments in order to support the concurrent execution of multiple communicating tasks in a multi-agent ecosystem
- Relaxing the requirement of a single shared data space to allow for hybrid workflow executions on top of multi-cloud or hybrid cloud/HPC infrastructures.
StreamFlow is freely available on GitHub, PyPI, and Docker Hub (alphaunito/streamflow) under the LGPLv3 license. It has been selected as an exploring technology by the EC Innovation Radar.
I. Colonnelli, B. Cantalupo, I. Merelli, and M. Aldinucci, “StreamFlow: cross-breeding cloud with HPC”, IEEE Transactions on Emerging Topics in Computing, 2020. DOI: 10.1109/TETC.2020.3019202
S. Leo, I. Colonnelli, M. Aldinucci et al., “Recording provenance of workflow runs with RO-Crate”, PLoS ONE, vol. 19(9), pp. 1–35, 2024.
I. Colonnelli et al., “Cross-Facility Federated Learning”, Procedia Computer Science, vol. 240, pp. 3–12, 2024.
2026
2026
The Italian pathfinding strategy for RISC-V-based HPC: Progress within DARE-SGA1 project
Giovanni Agosta, Marco Aldinucci, Marcello Barbirotta, Andrea Bartolini, Laura Bellentani, Andrea Biagioni, Daniele Cattaneo, Daniele Cesarini, Carlotta Chiarini, Iacopo Colonnelli, Francesco Conti, Pietro Delugas, Lev Denisov, Ottorino Frezza, Angelo Garofalo, Marco Grangetto, Juan David Guerrero Balaguera, Francesca Lo Cicero, Alessandro Lonardo, Michele Martinelli, Andrea Maslov, Francesco Menichelli, Mauro Olivieri, Pierpaolo Perticaroli, Luca Pontisso, Cristian Rossi, Josie Esteban Rodriguez Condia, Daniele Rossi, Davide Rossi, Sergio Saponara, Antonio Sciarappa, Francesco Simula, Matteo Sonza Reorda, Gabriele Spadaro, Massimo Torquati, Piero Vicini
Microprocessors and Microsystems
capio dare jupyter-workflow streamflow
Driven by the need of technological sovereignty, the European Union has accelerated the development of a sustainable High-Performance Computing (HPC) ecosystem. In this perspective, the DARE SGA1 (Digital Autonomy with RISC-V in Europe) project seeks to decouple European infrastructure from proprietary, non-European hardware by developing high-performance processors and computing accelerators based on the open-source RISC-V architecture. This paper introduces and reports about the current status of the multifaceted contributions of Italian partners to the critical pathfinding activities within the DARE SGA1 framework aiming at closing the gap between current prototypes and future HPC systems as well as establishing a bridge towards a fully independent European supercomputing supply chain. In particular, the activities focus on (i) novel RISC-V-based accelerator designs optimized for energy efficiency and high throughput and scaling-out capability; (ii) improvement of RISC-V-based system reliability and resiliency; (iii) evolution of the software stack for low level software and compiler optimizations (iv) co-design through benchmarking with AI and Quantum Chemistry scalable applications.
@article{2026:micpro:dare,
abstract = {Driven by the need of technological sovereignty, the European Union has accelerated the development of a sustainable High-Performance Computing (HPC) ecosystem. In this perspective, the DARE SGA1 (Digital Autonomy with RISC-V in Europe) project seeks to decouple European infrastructure from proprietary, non-European hardware by developing high-performance processors and computing accelerators based on the open-source RISC-V architecture. This paper introduces and reports about the current status of the multifaceted contributions of Italian partners to the critical pathfinding activities within the DARE SGA1 framework aiming at closing the gap between current prototypes and future HPC systems as well as establishing a bridge towards a fully independent European supercomputing supply chain. In particular, the activities focus on (i) novel RISC-V-based accelerator designs optimized for energy efficiency and high throughput and scaling-out capability; (ii) improvement of RISC-V-based system reliability and resiliency; (iii) evolution of the software stack for low level software and compiler optimizations (iv) co-design through benchmarking with AI and Quantum Chemistry scalable applications.},
author = {Giovanni Agosta and Marco Aldinucci and Marcello Barbirotta and Andrea Bartolini and Laura Bellentani and Andrea Biagioni and Daniele Cattaneo and Daniele Cesarini and Carlotta Chiarini and Iacopo Colonnelli and Francesco Conti and Pietro Delugas and Lev Denisov and Ottorino Frezza and Angelo Garofalo and Marco Grangetto and Juan David Guerrero Balaguera and Francesca {Lo Cicero} and Alessandro Lonardo and Michele Martinelli and Andrea Maslov and Francesco Menichelli and Mauro Olivieri and Pierpaolo Perticaroli and Luca Pontisso and Cristian Rossi and Josie Esteban Rodriguez Condia and Daniele Rossi and Davide Rossi and Sergio Saponara and Antonio Sciarappa and Francesco Simula and Matteo Sonza Reorda and Gabriele Spadaro and Massimo Torquati and Piero Vicini},
doi = {10.1016/j.micpro.2026.105321},
issn = {0141-9331},
journal = {Microprocessors and Microsystems},
keywords = {dare, streamflow, jupyter-workflow, capio},
pages = {105321},
title = {The Italian pathfinding strategy for {RISC-V}-based {HPC}: Progress within {DARE-SGA1} project},
url = {https://iris.unito.it/retrieve/e6b5b8a2-a130-4123-9f3b-22f2befeadd0/DARE_ICSC_IEEE_MICPRO.pdf},
volume = {124},
year = 2026,
bdsk-url-1 = {https://iris.unito.it/retrieve/e6b5b8a2-a130-4123-9f3b-22f2befeadd0/DARE_ICSC_IEEE_MICPRO.pdf},
bdsk-url-2 = {https://doi.org/10.1016/j.micpro.2026.105321}} Three ways to share a QPU: Scheduling strategies for hybrid Quantum-HPC applications
Marco Cipollini, Simone Rizzo, Sergio Iserte, Paolo Viviani, Giacomo Vitali, Matteo Barbieri, Gabriella Bettonte, Elisabetta Boella, Fulvio Ganz, Roberto Rocco, Orazio Spina, Antonio J. Peña, Petter Sandås, Iacopo Colonnelli, Alberto Scionti, Chiara Vercellino, Emanuele Dri, Jonathan Frassineti, Sara Marzella, Andrea Muratori, Daniele Ottaviani, Olivier Terzo, Bartolomeo Montrucchio, Daniele Gregori
Future Generation Computer Systems
icsc streamflow
As quantum computing (QC) technologies mature, their integration into established high-performance computing (HPC) infrastructures is becoming a central objective for next-generation computing systems. However, unlocking the potential of hybrid platforms for computationally demanding workloads remains challenging. The mismatch between quantum and classical programming models, the limited maturity of quantum software stacks, and the scarcity of quantum processing units (QPUs) above all, necessitate scheduling strategies that go beyond standard HPC mechanisms to manage such heterogeneous and constrained resources. To address this issue, we investigate three distinct methodologies for HPC-QC resource scheduling: time-based multiplexing, dynamic resource management, and workflow decomposition. Experimental validation on production HPC clusters and real quantum hardware demonstrates the effectiveness of these approaches under different workload scenarios. Malleability and workflow strategies significantly optimise classical resource utilisation, reducing consumption by up to 45.7% and 64% respectively, proving to be best fitted for hybrid jobs where quantum and classical workloads are evenly balanced. Conversely, time-multiplexing enhances QPU utilisation and reduces execution time at the cluster level, making it the optimal strategy for the opposite context, which is characterised by high classical-quantum workload imbalances. These findings underscore the practical viability of tailored scheduling strategies for hybrid HPC-QC environments and highlight their complementarity in building efficient, scalable software stacks for next-generation quantum-accelerated facilities.
@article{2026:fgcs:qpu,
abstract = {As quantum computing (QC) technologies mature, their integration into established high-performance computing (HPC) infrastructures is becoming a central objective for next-generation computing systems. However, unlocking the potential of hybrid platforms for computationally demanding workloads remains challenging. The mismatch between quantum and classical programming models, the limited maturity of quantum software stacks, and the scarcity of quantum processing units (QPUs) above all, necessitate scheduling strategies that go beyond standard HPC mechanisms to manage such heterogeneous and constrained resources. To address this issue, we investigate three distinct methodologies for HPC-QC resource scheduling: time-based multiplexing, dynamic resource management, and workflow decomposition. Experimental validation on production HPC clusters and real quantum hardware demonstrates the effectiveness of these approaches under different workload scenarios. Malleability and workflow strategies significantly optimise classical resource utilisation, reducing consumption by up to 45.7\% and 64\% respectively, proving to be best fitted for hybrid jobs where quantum and classical workloads are evenly balanced. Conversely, time-multiplexing enhances QPU utilisation and reduces execution time at the cluster level, making it the optimal strategy for the opposite context, which is characterised by high classical-quantum workload imbalances. These findings underscore the practical viability of tailored scheduling strategies for hybrid HPC-QC environments and highlight their complementarity in building efficient, scalable software stacks for next-generation quantum-accelerated facilities.},
author = {Marco Cipollini and Simone Rizzo and Sergio Iserte and Paolo Viviani and Giacomo Vitali and Matteo Barbieri and Gabriella Bettonte and Elisabetta Boella and Fulvio Ganz and Roberto Rocco and Orazio Spina and Antonio J. Pe\~{n}a and Petter Sand\r{a}s and Iacopo Colonnelli and Alberto Scionti and Chiara Vercellino and Emanuele Dri and Jonathan Frassineti and Sara Marzella and Andrea Muratori and Daniele Ottaviani and Olivier Terzo and Bartolomeo Montrucchio and Daniele Gregori},
doi = {10.1016/j.future.2026.108699},
issn = {0167-739X},
journal = {Future Generation Computer Systems},
keywords = {streamflow, icsc},
pages = {108699},
title = {Three ways to share a QPU: Scheduling strategies for hybrid Quantum-HPC applications},
url = {https://iris.unito.it/retrieve/e8086c96-8b84-47b7-bc9a-a43a34de633d/2604.14955v2.pdf},
volume = {185},
year = 2026,
bdsk-url-1 = {https://iris.unito.it/retrieve/e8086c96-8b84-47b7-bc9a-a43a34de633d/2604.14955v2.pdf},
bdsk-url-2 = {https://doi.org/10.1016/j.future.2026.108699}} Hybrid Streaming Workflows for Scalable Visualization in Astronomy and Cosmology
Eva Sciacca, Nicola Tuccari, Marco Edoardo Santimaria, Valentina Cesare, Fabio Vitello, Alberto Mulone, Doriana Medić, Iacopo Colonnelli
SN Computer Science
icsc space streamflow
Modern astrophysics and cosmology generate petabyte-scale datasets that demand advanced tools for scalable visualization and analysis. The Visualization Interface for the Virtual Observatory (VisIVO) provides multi-dimensional data exploration, but its integration with heterogeneous computing infrastructures and data-intensive workflows has remained challenging. Here we present a portable and reproducible approach that combines VisIVO with the StreamFlow workflow engine and the CAPIO middleware. StreamFlow enables hybrid execution across cloud-HPC environments, while CAPIO injects transparent I/O streaming into legacy file-based pipelines without modifying application code. Together, these technologies accelerate high–performance visualization, improve reproducibility, and reduce I/O bottlenecks. We demonstrate our approach on large-scale cosmological simulations produced with ChaNGa and GADGET-based codes, including studies of massive neutrinos in the DEMNUni suite. Our results show that streaming-enhanced workflows deliver up to 50% performance gains and enable interactive, scalable visualization across distributed infrastructures, paving the way for exascale-ready scientific discovery.
@article{2026:sn:visivo,
abstract = {Modern astrophysics and cosmology generate petabyte-scale datasets that demand advanced tools for scalable visualization and analysis. The Visualization Interface for the Virtual Observatory (VisIVO) provides multi-dimensional data exploration, but its integration with heterogeneous computing infrastructures and data-intensive workflows has remained challenging. Here we present a portable and reproducible approach that combines VisIVO with the StreamFlow workflow engine and the CAPIO middleware. StreamFlow enables hybrid execution across cloud-HPC environments, while CAPIO injects transparent I/O streaming into legacy file-based pipelines without modifying application code. Together, these technologies accelerate high--performance visualization, improve reproducibility, and reduce I/O bottlenecks. We demonstrate our approach on large-scale cosmological simulations produced with ChaNGa and GADGET-based codes, including studies of massive neutrinos in the DEMNUni suite. Our results show that streaming-enhanced workflows deliver up to 50{\%} performance gains and enable interactive, scalable visualization across distributed infrastructures, paving the way for exascale-ready scientific discovery.},
author = {Sciacca, Eva and Tuccari, Nicola and Santimaria, Marco Edoardo and Cesare, Valentina and Vitello, Fabio and Mulone, Alberto and Medi{\'{c}}, Doriana and Colonnelli, Iacopo},
doi = {10.1007/s42979-026-05076-4},
issn = {2661-8907},
journal = {SN Computer Science},
keywords = {streamflow, space, icsc},
month = jun,
number = 5,
pages = 527,
title = {Hybrid Streaming Workflows for Scalable Visualization in Astronomy and Cosmology},
url = {https://link.springer.com/content/pdf/10.1007/s42979-026-05076-4.pdf},
volume = 7,
year = {2026},
bdsk-url-1 = {https://link.springer.com/content/pdf/10.1007/s42979-026-05076-4.pdf},
bdsk-url-2 = {https://doi.org/10.1007/s42979-026-05076-4}} A formal framework for fault tolerance in hybrid scientific workflows
Alberto Mulone, Doriana Medić, Iacopo Colonnelli, Marco Aldinucci
Future Generation Computer Systems
eupex icsc space streamflow
In large-scale distributed systems, failures are routine events whose occurrences increase with the number of computational tasks and execution locations. The advantage of representing an application as a workflow is the possibility of exploiting Workflow Management System (WMS) features such as portability, scalability, and, crucially, reliability. Among these, reliability is essential for ensuring robust execution in dynamic and failure-prone environments. In recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. Consequently, the number of possible points of failure during the execution increased, creating a need for sophisticated fault tolerance mechanisms capable of addressing the specific requirements of hybrid systems. This work introduces a formal framework for a fault tolerance mechanism in hybrid workflows, enabling failure recovery through a rollback approach. The framework is rigorously defined by adapting and extending an existing workflow semantics tailored for hybrid execution. Our method leverages provenance data from workflow execution up to the point of failure, and creates a recovery workflow that spans multiple infrastructures. The rollback approach provides a robust and reliable strategy to ensure resilience against step failures and potential data loss. We then implement this mechanism in the StreamFlow WMS, and evaluate it using two case studies: the 1000 Genomes workflow and a synthetic workflow featuring iterative patterns. Experiments showcase the conceptual validity of our approach and assess the overhead introduced by the mechanism, including data availability checks.
@article{25:fgcs:ft,
abstract = {In large-scale distributed systems, failures are routine events whose occurrences increase with the number of computational tasks and execution locations. The advantage of representing an application as a workflow is the possibility of exploiting Workflow Management System (WMS) features such as portability, scalability, and, crucially, reliability. Among these, reliability is essential for ensuring robust execution in dynamic and failure-prone environments. In recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. Consequently, the number of possible points of failure during the execution increased, creating a need for sophisticated fault tolerance mechanisms capable of addressing the specific requirements of hybrid systems. This work introduces a formal framework for a fault tolerance mechanism in hybrid workflows, enabling failure recovery through a rollback approach. The framework is rigorously defined by adapting and extending an existing workflow semantics tailored for hybrid execution. Our method leverages provenance data from workflow execution up to the point of failure, and creates a recovery workflow that spans multiple infrastructures. The rollback approach provides a robust and reliable strategy to ensure resilience against step failures and potential data loss. We then implement this mechanism in the StreamFlow WMS, and evaluate it using two case studies: the 1000 Genomes workflow and a synthetic workflow featuring iterative patterns. Experiments showcase the conceptual validity of our approach and assess the overhead introduced by the mechanism, including data availability checks.},
author = {Alberto Mulone and Doriana Medi\'{c} and Iacopo Colonnelli and Marco Aldinucci},
doi = {10.1016/j.future.2025.108188},
issn = {0167-739X},
journal = {{F}uture {G}eneration {C}omputer {S}ystems},
keywords = {icsc, eupex, space, streamflow},
pages = {108188},
title = {A formal framework for fault tolerance in hybrid scientific workflows},
url = {https://iris.unito.it/retrieve/b9adc822-4455-4fe1-a8c1-8a66c084335e/1-s2.0-S0167739X25004820-main.pdf},
volume = {176},
year = {2026},
bdsk-url-1 = {https://iris.unito.it/retrieve/b9adc822-4455-4fe1-a8c1-8a66c084335e/1-s2.0-S0167739X25004820-main.pdf},
bdsk-url-2 = {https://doi.org/10.1016/j.future.2025.108188}} A terminology for scientific workflow systems
Frédéric Suter, Tainã Coleman, İlkay Altintaş, Rosa M. Badia, Bartosz Balis, Kyle Chard, Iacopo Colonnelli, Ewa Deelman, Paolo Di Tommaso, Thomas Fahringer, Carole Goble, Shantenu Jha, Daniel S. Katz, Johannes K"oster, Ulf Leser, Kshitij Mehta, Hilary Oliver, J.-Luc Peterson, Giovanni Pizzi, Lo"ic Pottier, Ra"ul Sirvent, Eric Suchyta, Douglas Thain, Sean R. Wilkinson, Justin M. Wozniak, Rafael Ferreira da Silva
Future Generation Computer Systems
streamflow
The term “scientific workflow” has evolved over the last two decades to encompass a broad range of compositions of interdependent compute tasks and data movements. It has also become an umbrella term for processing in modern scientific applications. Today, many scientific applications can be considered as workflows made of multiple dependent steps, and hundreds of workflow systems have been developed to manage and run these scientific workflows. However, no turnkey solution has emerged from the field to address the diversity of scientific processes and the infrastructure on which they are supposed to be implemented. Instead, new research problems requiring the execution of scientific workflows with some novel feature often lead to the development of an entirely new workflow system. A direct consequence of this situation is that many existing workflow management systems (WMSs) share some salient features, offer similar functionalities, and can manage the same categories of workflows but at the same time also have some distinct capabilities that can be important for specific applications. This situation makes researchers who develop workflows face the complex question of selecting a WMS. This selection can be driven by technical considerations, to find the system that is the most appropriate for their application and for the computing and storage resources available to them, or other factors such as reputation, adoption, strong community support, or long-term sustainability. To address this problem, a group of WMS developers and practitioners joined their efforts to produce a community-based terminology of WMSs. This paper summarizes their findings and introduces this new terminology to characterize WMSs. This terminology is composed of fives axes: workflow structure and characteristics, composition, orchestration, data management, and metadata capture. Each axis comprises several concepts that capture the prominent features of WMSs. Based on this terminology, this paper also presents a classification of 23 existing WMSs according to the proposed axes and terms.
@article{26:FGCS:Terminology,
abstract = {The term ``scientific workflow'' has evolved over the last two decades to encompass a broad range of compositions of interdependent compute tasks and data movements. It has also become an umbrella term for processing in modern scientific applications. Today, many scientific applications can be considered as workflows made of multiple dependent steps, and hundreds of workflow systems have been developed to manage and run these scientific workflows. However, no turnkey solution has emerged from the field to address the diversity of scientific processes and the infrastructure on which they are supposed to be implemented. Instead, new research problems requiring the execution of scientific workflows with some novel feature often lead to the development of an entirely new workflow system. A direct consequence of this situation is that many existing workflow management systems (WMSs) share some salient features, offer similar functionalities, and can manage the same categories of workflows but at the same time also have some distinct capabilities that can be important for specific applications. This situation makes researchers who develop workflows face the complex question of selecting a WMS. This selection can be driven by technical considerations, to find the system that is the most appropriate for their application and for the computing and storage resources available to them, or other factors such as reputation, adoption, strong community support, or long-term sustainability. To address this problem, a group of WMS developers and practitioners joined their efforts to produce a community-based terminology of WMSs. This paper summarizes their findings and introduces this new terminology to characterize WMSs. This terminology is composed of fives axes: workflow structure and characteristics, composition, orchestration, data management, and metadata capture. Each axis comprises several concepts that capture the prominent features of WMSs. Based on this terminology, this paper also presents a classification of 23 existing WMSs according to the proposed axes and terms.},
author = {Suter, Fr\'{e}d\'{e}ric and Coleman, Tain\~{a} and Altinta\c{s}, \.{I}lkay and Badia, Rosa M. and Balis, Bartosz and Chard, Kyle and Colonnelli, Iacopo and Deelman, Ewa and Di Tommaso, Paolo and Fahringer, Thomas and Goble, Carole and Jha, Shantenu and Katz, Daniel S. and K"{o}ster, Johannes and Leser, Ulf and Mehta, Kshitij and Oliver, Hilary and Peterson, J.-Luc and Pizzi, Giovanni and Pottier, Lo"{i}c and Sirvent, Ra"{u}l and Suchyta, Eric and Thain, Douglas and R. Wilkinson, Sean and Wozniak, Justin M. and Ferreira da Silva, Rafael},
doi = {10.1016/j.future.2025.107974},
issn = {0167-739X},
journal = {Future Generation Computer Systems},
keywords = {streamflow},
pages = {107974},
title = {A terminology for scientific workflow systems},
url = {https://www.arxiv.org/abs/2506.07838},
volume = {174},
year = {2026},
bdsk-url-1 = {https://www.arxiv.org/abs/2506.07838},
bdsk-url-2 = {https://doi.org/10.1016/j.future.2025.107974}} Towards User-Friendly HPC: A Web Platform for Hybrid Workflows
Tommaso Foglio Bonda, Ivan Lusso, Doriana Medić, Alessia Antelmi, Eva Sciacca, Nicola Tuccari, Fabio Vitello
Proceedings of the 18th International Workshop on Science Gateways (IWSG 2026)
hpc4ai space streamflow visivo
@inproceedings{2026:iwsg:fogliobonda,
title = {Towards User-Friendly {HPC}: A Web Platform for Hybrid Workflows},
author = {Foglio Bonda, Tommaso and Lusso, Ivan and Medi{\'c}, Doriana and Antelmi, Alessia and Sciacca, Eva and Tuccari, Nicola and Vitello, Fabio},
year = 2026,
month = jun,
booktitle = {Proceedings of the 18th International Workshop on Science Gateways (IWSG 2026)},
address = {Catania, Italy},
keywords = {streamflow, hpc4ai, space, visivo}
} A Privacy-Preserving Hybrid Machine Learning Pipeline for Regulated Environments: Architecture and Evaluation
Roberto Bagnato, Doriana Medić, Luca Vollero, Anna Sabatini, Tommaso Foglio Bonda, Manuele Innocenti, Marco Aldinucci
2026 IEEE International Conference on Cyber Security and Resilience (CSR) Workshops (2P-DPA)
confidential hpc4ai streamflow
@inproceedings{2026:csr:bagnato,
title = {A Privacy-Preserving Hybrid Machine Learning Pipeline for Regulated Environments: Architecture and Evaluation},
author = {Bagnato, Roberto and Medi{\'c}, Doriana and Vollero, Luca and Sabatini, Anna and Foglio Bonda, Tommaso and Innocenti, Manuele and Aldinucci, Marco},
year = 2026,
month = aug,
booktitle = {2026 IEEE International Conference on Cyber Security and Resilience (CSR) Workshops (2P-DPA)},
publisher = {IEEE},
address = {Lisbon, Portugal},
keywords = {streamflow, hpc4ai, confidential}
} 2025
2025
Towards RISC-V-based HPC: The Italian Pathfinding Activities in the DARE-SGA1 Project
Giovanni Agosta, Marco Aldinucci, Andrea Bartolini, Laura Bellentani, Andrea Biagioni, Daniele Cesarini, Carlotta Chiarini, Iacopo Colonnelli, Pietro Delugas, Lev Denisov, Ottorino Frezza, Marco Grangetto, Francesca Lo Cicero, Alessandro Lonardo, Michele Martinelli, Andrea Maslov, Mauro Olivieri, Pierpaolo Perticaroli, Luca Pontisso, Cristian Rossi, Davide Rossi, Sergio Saponara, Antonio Sciarappa, Francesco Simula, Matteo Sonza Reorda, Massimo Torquati, Piero Vicini
28th Euromicro Conference on Digital System Design, DSD 2025
capio dare streamflow
The European Union's efforts towards technological sovereignty in High-Performance Computing are driving research and development of RISC-V-based supercomputers. The DARE SGA1 project, in particular, aims to develop chips designed and owned by Europeans. This paper introduces the Italian contribution to DARE SGA1 regarding pathfinding activities toward future RISC-V-based accelerator designs, reliability improvements, system software, and AI and Quantum Chemistry applications.
@inproceedings{2025:dsd:dare,
abstract = {The European Union's efforts towards technological sovereignty in High-Performance Computing are driving research and development of RISC-V-based supercomputers. The DARE SGA1 project, in particular, aims to develop chips designed and owned by Europeans. This paper introduces the Italian contribution to DARE SGA1 regarding pathfinding activities toward future RISC-V-based accelerator designs, reliability improvements, system software, and AI and Quantum Chemistry applications.},
author = {Agosta, Giovanni and Aldinucci, Marco and Bartolini, Andrea and Bellentani, Laura and Biagioni, Andrea and Cesarini, Daniele and Chiarini, Carlotta and Colonnelli, Iacopo and Delugas, Pietro and Denisov, Lev and Frezza, Ottorino and Grangetto, Marco and Lo Cicero, Francesca and Lonardo, Alessandro and Martinelli, Michele and Maslov, Andrea and Olivieri, Mauro and Perticaroli, Pierpaolo and Pontisso, Luca and Rossi, Cristian and Rossi, Davide and Saponara, Sergio and Sciarappa, Antonio and Simula, Francesco and Sonza Reorda, Matteo and Torquati, Massimo and Vicini, Piero},
booktitle = {28th Euromicro Conference on Digital System Design, {DSD} 2025},
doi = {10.1109/DSD67783.2025.00057},
keywords = {dare, streamflow, capio},
location = {Salerno, Italy},
mon = {sep},
pages = {360--367},
publisher = {{IEEE}},
title = {Towards {RISC-V}-based {HPC:} The Italian Pathfinding Activities in the {DARE-SGA1} Project},
url = {https://iris.unito.it/retrieve/03658af6-782d-4f9c-9108-b0fc1f1859eb/DARE_ICSC_EPDSD2025-1.pdf},
year = {2025},
bdsk-url-1 = {https://iris.unito.it/retrieve/03658af6-782d-4f9c-9108-b0fc1f1859eb/DARE_ICSC_EPDSD2025-1.pdf},
bdsk-url-2 = {https://doi.org/10.1109/DSD67783.2025.00057}} Dynamic Solutions for Hybrid Quantum-HPC Resource Allocation
Roberto Rocco, Simone Rizzo, Matteo Barbieri, Gabriella Bettonte, Elisabetta Boella, Fulvio Ganz, Sergio Iserte, Antonio J. Peña, Petter Sandås, Alberto Scionti, Olivier Terzo, Chiara Vercellino, Giacomo Vitali, Paolo Viviani, Jonathan Frassineti, Sara Marzella, Daniele Ottaviani, Iacopo Colonnelli, Daniele Gregori
2025 IEEE International Conference on Quantum Computing and Engineering (QCE)
icsc streamflow
The integration of quantum computers within classical High-Performance Computing (HPC) infrastructures is receiving increasing attention, with the former expected to serve as accelerators for specific computational tasks. However, combining HPC and quantum computers presents significant technical challenges, including resource allocation. This paper presents a novel malleability-based approach, alongside a workflow-based strategy, to optimize resource utilization in hybrid HPC-quantum workloads. With both these approaches, we can release classical resources when computations are offloaded to the quantum computer and reallocate them once quantum processing is complete. Our experiments with a hybrid HPC-quantum use case show the benefits of dynamic allocation, highlighting the potential of those solutions.
@inproceedings{2025:qce:hybrid,
abstract = {The integration of quantum computers within classical High-Performance Computing (HPC) infrastructures is receiving increasing attention, with the former expected to serve as accelerators for specific computational tasks. However, combining HPC and quantum computers presents significant technical challenges, including resource allocation. This paper presents a novel malleability-based approach, alongside a workflow-based strategy, to optimize resource utilization in hybrid HPC-quantum workloads. With both these approaches, we can release classical resources when computations are offloaded to the quantum computer and reallocate them once quantum processing is complete. Our experiments with a hybrid HPC-quantum use case show the benefits of dynamic allocation, highlighting the potential of those solutions.},
author = {Rocco, Roberto and Rizzo, Simone and Barbieri, Matteo and Bettonte, Gabriella and Boella, Elisabetta and Ganz, Fulvio and Iserte, Sergio and Pe{\~{n}}a, Antonio J. and Sand{\aa}s, Petter and Scionti, Alberto and Terzo, Olivier and Vercellino, Chiara and Vitali, Giacomo and Viviani, Paolo and Frassineti, Jonathan and Marzella, Sara and Ottaviani, Daniele and Colonnelli, Iacopo and Gregori, Daniele},
booktitle = {2025 {IEEE} International Conference on Quantum Computing and Engineering (QCE)},
doi = {10.1109/QCE65121.2025.10289},
keywords = {streamflow, icsc},
location = {Albuquerque, NM, USA},
month = sep,
pages = {34--40},
publisher = {{IEEE}},
title = {Dynamic Solutions for Hybrid Quantum-HPC Resource Allocation},
url = {https://iris.unito.it/retrieve/47e89e49-136a-48ff-beeb-63cee8bdceab/Preprint.pdf},
year = {2025},
bdsk-url-1 = {https://iris.unito.it/retrieve/47e89e49-136a-48ff-beeb-63cee8bdceab/Preprint.pdf},
bdsk-url-2 = {https://doi.org/10.1109/QCE65121.2025.10289}} EuroHPC SPACE CoE: Redesigning Scalable Parallel Astrophysical Codes for Exascale. Invited Paper
Nitin Shukla, Alessandro Romeo, Caterina Caravita, Lubomir Riha, Ondrej Vysocky, Petr Strakos, Milan Jaros, João Barbosa, Radim Vavrı́k, Andrea Mignone, Marco Rossazza, Stefano Truzzi, Vittoria Berta, Iacopo Colonnelli, Doriana Medić, Elisabetta Boella, Daniele Gregori, Eva Sciacca, Luca Tornatore, Giuliano Taffoni, Pranab J. Deka, Fabio Bacchini, Rostislav-Paul Wilhelm, Georgios Doulis, Khalil Pierre, Luciano Rezzolla, Tine Colman, Benoı̂t Commerçon, Othman Bouizi, Matthieu Kuhn, Erwan Raffin, Marc Sergent, Robert Wissing, Guillermo Marin, Klaus Dolag, Geray S. Karademir, Gino Perna, Marisa Zanotti, Sebastian Trujillo-Gomez
Proceedings of the 22nd ACM International Conference on Computing Frontiers: Workshops and Special Sessions, CF 2025, Cagliari, Italy, May 28-30, 2025
space streamflow
High Performance Computing (HPC) based simulations are crucial in Astrophysics & Cosmology (A&C), helping scientists investigate and understand complex astrophysical phenomena. Taking advantage of exascale computing capabilities is essential for these efforts. However, the unprecedented architectural complexity of exascale systems impacts legacy codes. The SPACE Centre of Excellence (CoE) aims to re-engineer key astrophysical codes to tackle new computational challenges by adopting innovative programming paradigms and software (SW) solutions. SPACE brings together scientists, code developers, HPC experts, hardware (HW) manufacturers, and SW developers. This collaboration enhances exascale A&C applications, promoting the use of exascale and post-exascale computing capabilities. Additionally, SPACE addresses high-performance data analysis for the massive data outputs from exascale simulations and modern observations, using machine learning (ML) and visualisation tools. The project facilitates application deployment across platforms by focusing on code repositories and data sharing, integrating European astrophysical communities around exascale computing with standardised SW and data protocols.
@inproceedings{25:cf:space,
abstract = {High Performance Computing (HPC) based simulations are crucial in Astrophysics \& Cosmology (A\&C), helping scientists investigate and understand complex astrophysical phenomena. Taking advantage of exascale computing capabilities is essential for these efforts. However, the unprecedented architectural complexity of exascale systems impacts legacy codes. The SPACE Centre of Excellence (CoE) aims to re-engineer key astrophysical codes to tackle new computational challenges by adopting innovative programming paradigms and software (SW) solutions. SPACE brings together scientists, code developers, HPC experts, hardware (HW) manufacturers, and SW developers. This collaboration enhances exascale A&C applications, promoting the use of exascale and post-exascale computing capabilities. Additionally, SPACE addresses high-performance data analysis for the massive data outputs from exascale simulations and modern observations, using machine learning (ML) and visualisation tools. The project facilitates application deployment across platforms by focusing on code repositories and data sharing, integrating European astrophysical communities around exascale computing with standardised SW and data protocols.},
author = {Shukla, Nitin and Romeo, Alessandro and Caravita, Caterina and Riha, Lubomir and Vysocky, Ondrej and Strakos, Petr and Jaros, Milan and Barbosa, Jo{\~{a}}o and Vavr{\'{\i}}k, Radim and Mignone, Andrea and Rossazza, Marco and Truzzi, Stefano and Berta, Vittoria and Colonnelli, Iacopo and Medi\'{c}, Doriana and Boella, Elisabetta and Gregori, Daniele and Sciacca, Eva and Tornatore, Luca and Taffoni, Giuliano and Deka, Pranab J. and Bacchini, Fabio and Wilhelm, Rostislav{-}Paul and Doulis, Georgios and Pierre, Khalil and Rezzolla, Luciano and Colman, Tine and Commer{\c{c}}on, Beno{\^{\i}}t and Bouizi, Othman and Kuhn, Matthieu and Raffin, Erwan and Sergent, Marc and Wissing, Robert and Marin, Guillermo and Dolag, Klaus and Karademir, Geray S. and Perna, Gino and Zanotti, Marisa and Trujillo{-}Gomez, Sebastian},
booktitle = {Proceedings of the 22nd {ACM} International Conference on Computing Frontiers: Workshops and Special Sessions, {CF} 2025, Cagliari, Italy, May 28-30, 2025},
doi = {10.1145/3706594.3728892},
editor = {Francesca Palumbo and Antonino Tumeo and Ana Lucia Varbanescu and Yogesh Simmhan},
keywords = {streamflow, space},
location = {Cagliari, Italy},
pages = {177--184},
publisher = {{ACM}},
title = {{EuroHPC} {SPACE} {CoE}: Redesigning Scalable Parallel Astrophysical Codes for Exascale. Invited Paper},
url = {https://iris.unito.it/retrieve/f0d21f94-0973-4714-89f5-056e73b2356b/SpaceCF2025Preprint.pdf},
year = {2025},
bdsk-url-1 = {https://iris.unito.it/retrieve/f0d21f94-0973-4714-89f5-056e73b2356b/SpaceCF2025Preprint.pdf},
bdsk-url-2 = {https://doi.org/10.1145/3706594.3728892}} The Cloud-HPC infrastructure for Hazard Mapping and vulnerability Monitoring (HaMMon)
Mauro Imbrosciano, Eva Sciacca, Fabio Vitello, Leonardo Pelonero, Francesco Franchina, Ugo Becciani, Iacopo Colonnelli, Doriana Medić
2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP)
hpc4ai icsc jupyter-workflow streamflow system-sw
The HaMMon project is the outcome of an industrial partnership that includes many Italian research institutions and private companies. It is led by UnipolSai and Leitha, and funded by the ICSC, the Italian National Research Center for High Performance Computing, Big Data and Quantum Computing.The ambition of HaMMon is to build a flexible and scalable platform to analyze the hydrogeological and atmospheric balance of the Italian territory. The project aims to expand the current knowledge in hazard mapping, monitoring, and forecasting from an industrial perspective by leveraging innovative technologies and the interdisciplinary activities carried out by the ICSC.In this work, we present the cloud-HPC infrastructure deployed in the High-Performance Computing for Artificial Intelligence (HPC4AI) green data center of the University of Turin which supports the testing and development of HaMMon's applications and services. We describe the current activities and preliminary results related to the integration of Photogrammetry techniques, Data Visualization and Artificial Intelligence technologies, applied on aerial images, to assess extreme natural events and evaluate their impact on risk-exposed assets.
@inproceedings{imbrosciano:2025:pdp,
abstract = {The HaMMon project is the outcome of an industrial partnership that includes many Italian research institutions and private companies. It is led by UnipolSai and Leitha, and funded by the ICSC, the Italian National Research Center for High Performance Computing, Big Data and Quantum Computing.The ambition of HaMMon is to build a flexible and scalable platform to analyze the hydrogeological and atmospheric balance of the Italian territory. The project aims to expand the current knowledge in hazard mapping, monitoring, and forecasting from an industrial perspective by leveraging innovative technologies and the interdisciplinary activities carried out by the ICSC.In this work, we present the cloud-HPC infrastructure deployed in the High-Performance Computing for Artificial Intelligence (HPC4AI) green data center of the University of Turin which supports the testing and development of HaMMon's applications and services. We describe the current activities and preliminary results related to the integration of Photogrammetry techniques, Data Visualization and Artificial Intelligence technologies, applied on aerial images, to assess extreme natural events and evaluate their impact on risk-exposed assets.},
author = {Mauro Imbrosciano and Eva Sciacca and Fabio Vitello and Leonardo Pelonero and Francesco Franchina and Ugo Becciani and Iacopo Colonnelli and Doriana Medi\'{c}},
booktitle = {2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP)},
date-modified = {2025-05-24 09:13:03 +0200},
doi = {10.1109/PDP66500.2025.00050},
issn = {2377-5750},
keywords = {system-sw, hpc4ai, icsc, jupyter-workflow, streamflow},
pages = {309--316},
publisher = {IEEE},
title = {The Cloud-HPC infrastructure for Hazard Mapping and vulnerability Monitoring (HaMMon)},
url = {https://iris.unito.it/retrieve/36a223d5-7134-4469-80a0-705c0060a5d4/PDP_2025_HaMMON.pdf},
year = {2025},
bdsk-url-1 = {https://iris.unito.it/retrieve/36a223d5-7134-4469-80a0-705c0060a5d4/PDP_2025_HaMMON.pdf},
bdsk-url-2 = {https://doi.org/10.1109/PDP66500.2025.00050}} High Performance Visualization for Astrophysics and Cosmology
Nicola Tuccari, Eva Sciacca, Fabio Vitello, Iacopo Colonnelli, Yolanda Becerra, Enric Sosa Cintero, Guillermo Marin, Milan Jaros, Lubomir Riha, Petr Strakos, Sebastian Trujillo-Gomez, Emiliano Tramontana, Robert Wissing
2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP)
icsc space streamflow
Modern Astrophysics and Cosmology (A&C) projects produce immense data volumes, necessitating advanced software tools for data access, storage, and analysis. Visualization Interface for the Virtual Observatory (VisIVO) is one such tool enabling multi-dimensional data analysis and knowledge discovery across complex astrophysical datasets. Leveraging containerization and virtualization, VisIVO has been deployed on various distributed computing platforms. Additionally, Blender, an open-source 3D suite, provides robust tools for rendering and processing volumetric data, making it suitable for visualizing complex datasets. At the SPACE Center of Excellence these tools are being adapted for high-performance visualization of cosmological simulations performed with GADGET and ChaNGa on pre-exascale systems. However, implementing high-performance visualization on diverse HPC platforms presents several challenges, including hardware and software compatibility, data management, scalability, performance portability, and efficient resource allocation. This paper outlines strategies to integrate VisIVO with workflow frameworks and streaming platforms to address these challenges. Workflow frameworks enhance portability, scheduling, and reproducibility of visualization workflows on pre-exascale systems used in A&C simulations. We also discuss the use of streaming platforms to enable concurrent (i.e. in-situ) analysis and visualization of simulations, reducing the need to store full simulation data by leveraging distributed databases that stream the output data in real time. Lastly, we present an adaptation of Blender to handle large-scale particle-based astrophysical data, offering high-quality visualization with interactive exploration capabilities.
@inproceedings{tuccari:2025:pdp,
abstract = {Modern Astrophysics and Cosmology (A\&C) projects produce immense data volumes, necessitating advanced software tools for data access, storage, and analysis. Visualization Interface for the Virtual Observatory (VisIVO) is one such tool enabling multi-dimensional data analysis and knowledge discovery across complex astrophysical datasets. Leveraging containerization and virtualization, VisIVO has been deployed on various distributed computing platforms. Additionally, Blender, an open-source 3D suite, provides robust tools for rendering and processing volumetric data, making it suitable for visualizing complex datasets. At the SPACE Center of Excellence these tools are being adapted for high-performance visualization of cosmological simulations performed with GADGET and ChaNGa on pre-exascale systems. However, implementing high-performance visualization on diverse HPC platforms presents several challenges, including hardware and software compatibility, data management, scalability, performance portability, and efficient resource allocation. This paper outlines strategies to integrate VisIVO with workflow frameworks and streaming platforms to address these challenges. Workflow frameworks enhance portability, scheduling, and reproducibility of visualization workflows on pre-exascale systems used in A\&C simulations. We also discuss the use of streaming platforms to enable concurrent (i.e. in-situ) analysis and visualization of simulations, reducing the need to store full simulation data by leveraging distributed databases that stream the output data in real time. Lastly, we present an adaptation of Blender to handle large-scale particle-based astrophysical data, offering high-quality visualization with interactive exploration capabilities.},
author = {Tuccari, Nicola and Sciacca, Eva and Vitello, Fabio and Colonnelli, Iacopo and Becerra, Yolanda and Cintero, Enric Sosa and Marin, Guillermo and Jaros, Milan and Riha, Lubomir and Strakos, Petr and Trujillo-Gomez, Sebastian and Tramontana, Emiliano and Wissing, Robert},
booktitle = {2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP)},
doi = {10.1109/PDP66500.2025.00069},
issn = {2377-5750},
keywords = {icsc, space, streamflow},
pages = {443--450},
publisher = {IEEE},
title = {High Performance Visualization for Astrophysics and Cosmology},
url = {https://iris.unito.it/retrieve/994fe11b-b282-4267-80c9-6d60821ba2cb/PDP2025___WP3_SPACE_Vis.pdf},
year = {2025},
bdsk-url-1 = {https://iris.unito.it/retrieve/994fe11b-b282-4267-80c9-6d60821ba2cb/PDP2025___WP3_SPACE_Vis.pdf},
bdsk-url-2 = {https://doi.org/10.1109/PDP66500.2025.00069}} Towards a European HPC/AI ecosystem: a community-driven report
Petr Taborsky, Iacopo Colonnelli, Krzysztof Kurowski, Rakesh Sarma, Niels Henrik Pontoppidan, Branislav Jansı́k, Nicki Skafte Detlefsen, Jens Egholm Pedersen, Rasmus Larsen, Lars Kai Hansen
Procedia Computer Science
streamflow
The rapid advancements in AI and Machine Learning necessitate a robust computational infrastructure to support cutting-edge research and industrial applications. From the academic and industrial AI community perspective, voiced in the recent ELISE project, the European AI platform is recommended to center around the EuroHPC growing ecosystem. It should be user-driven, easily accessible, powerful, and compliant with European regulations. AI-optimized and dedicated supercomputers for the European AI community are also coming, in addition to upgrading partitions of existing EuroHPC systems to 'AI enabled' stage. Related calls have been initiated in September 2024. Further, conventional EuroHPC systems are suggested to be extended with quantum computing, edge AI, and neuromorphic computing to cater to AI models deployed on network edge devices and sustainability in the long run. The challenges are presented in three case studies, ranging from training Transformers on HPC to LLMs trained federally across three different Euro HPC systems to recent results on hybrid classical-quantum application. This paper concludes with case studies results-informed next steps believed to benefit AI practitioners and the broader AI community.
@article{icolonne:eurohpc:24,
abstract = {The rapid advancements in AI and Machine Learning necessitate a robust computational infrastructure to support cutting-edge research and industrial applications. From the academic and industrial AI community perspective, voiced in the recent ELISE project, the European AI platform is recommended to center around the EuroHPC growing ecosystem. It should be user-driven, easily accessible, powerful, and compliant with European regulations. AI-optimized and dedicated supercomputers for the European AI community are also coming, in addition to upgrading partitions of existing EuroHPC systems to 'AI enabled' stage. Related calls have been initiated in September 2024. Further, conventional EuroHPC systems are suggested to be extended with quantum computing, edge AI, and neuromorphic computing to cater to AI models deployed on network edge devices and sustainability in the long run. The challenges are presented in three case studies, ranging from training Transformers on HPC to LLMs trained federally across three different Euro HPC systems to recent results on hybrid classical-quantum application. This paper concludes with case studies results-informed next steps believed to benefit AI practitioners and the broader AI community.},
address = {Amsterdam, Netherlands},
author = {Taborsky, Petr and Colonnelli, Iacopo and Kurowski, Krzysztof and Sarma, Rakesh and Pontoppidan, Niels Henrik and Jans{\'\i}k, Branislav and Detlefsen, Nicki Skafte and Egholm Pedersen, Jens and Larsen, Rasmus and Hansen, Lars Kai},
booktitle = {Proceedings of the Second EuroHPC user day},
doi = {10.1016/j.procs.2025.02.269},
issn = {1877-0509},
journal = {Procedia Computer Science},
keywords = {streamflow},
pages = {140--149},
publisher = {Elsevier},
title = {Towards a European {HPC}/{AI} ecosystem: a community-driven report},
url = {https://www.sciencedirect.com/science/article/pii/S1877050925006301},
volume = {255},
year = {2025},
bdsk-url-1 = {https://www.sciencedirect.com/science/article/pii/S1877050925006301},
bdsk-url-2 = {https://doi.org/10.1016/j.procs.2025.02.269}} Accelerating SWIRL Workflows: A High-Performance Rust Backend for Distributed Execution
Tommaso Foglio Bonda, Doriana Medić, Alberto Mulone, Marco Aldinucci
Euro-Par 2025: Parallel Processing Workshops (HiPES 2025)
eupex icsc streamflow swirl
@inproceedings{2025:hipes:fogliobonda,
title = {Accelerating {SWIRL} Workflows: A High-Performance {Rust} Backend for Distributed Execution},
author = {Foglio Bonda, Tommaso and Medi{\'c}, Doriana and Mulone, Alberto and Aldinucci, Marco},
year = 2025,
month = aug,
booktitle = {Euro-Par 2025: Parallel Processing Workshops (HiPES 2025)},
publisher = {Springer Nature Switzerland},
address = {Brescia, Italy},
series = {Lecture Notes in Computer Science},
volume = 16294,
keywords = {swirl, streamflow, icsc, eupex}
} 2024
2024
Recording provenance of workflow runs with RO-Crate
Simone Leo, Michael R. Crusoe, Laura Rodríguez-Navas, Ra"ul Sirvent, Alexander Kanitz, Paul De Geest, Rudolf Wittner, Luca Pireddu, Daniel Garijo, José M. Fernández, Iacopo Colonnelli, Matej Gallo, Tazro Ohta, Hirotaka Suetake, Salvador Capella-Gutierrez, Renske Wit, Bruno P. Kinoshita, Stian Soiland-Reyes
PLoS ONE
across eupex icsc streamflow
Recording the provenance of scientific computation results is key to the support of traceability, reproducibility and quality assessment of data products. Several data models have been explored to address this need, providing representations of workflow plans and their executions as well as means of packaging the resulting information for archiving and sharing. However, existing approaches tend to lack interoperable adoption across workflow management systems. In this work we present Workflow Run RO-Crate, an extension of RO-Crate (Research Object Crate) and Schema.org to capture the provenance of the execution of computational workflows at different levels of granularity and bundle together all their associated objects (inputs, outputs, code, etc.). The model is supported by a diverse, open community that runs regular meetings, discussing development, maintenance and adoption aspects. Workflow Run RO-Crate is already implemented by several workflow management systems, allowing interoperable comparisons between workflow runs from heterogeneous systems. We describe the model, its alignment to standards such as W3C PROV, and its implementation in six workflow systems. Finally, we illustrate the application of Workflow Run RO-Crate in two use cases of machine learning in the digital image analysis domain.
@article{24:pone:wfrunrocrate,
abstract = {Recording the provenance of scientific computation results is key to the support of traceability, reproducibility and quality assessment of data products. Several data models have been explored to address this need, providing representations of workflow plans and their executions as well as means of packaging the resulting information for archiving and sharing. However, existing approaches tend to lack interoperable adoption across workflow management systems. In this work we present Workflow Run RO-Crate, an extension of RO-Crate (Research Object Crate) and Schema.org to capture the provenance of the execution of computational workflows at different levels of granularity and bundle together all their associated objects (inputs, outputs, code, etc.). The model is supported by a diverse, open community that runs regular meetings, discussing development, maintenance and adoption aspects. Workflow Run RO-Crate is already implemented by several workflow management systems, allowing interoperable comparisons between workflow runs from heterogeneous systems. We describe the model, its alignment to standards such as W3C PROV, and its implementation in six workflow systems. Finally, we illustrate the application of Workflow Run RO-Crate in two use cases of machine learning in the digital image analysis domain.},
author = {Leo, Simone and Crusoe, Michael R. and Rodr\'{i}guez-Navas, Laura and Sirvent, Ra"{u}l and Kanitz, Alexander and De Geest, Paul and Wittner, Rudolf and Pireddu, Luca and Garijo, Daniel and Fern\'{a}ndez, Jos\'{e} M. and Colonnelli, Iacopo and Gallo, Matej and Ohta, Tazro and Suetake, Hirotaka and Capella-Gutierrez, Salvador and de Wit, Renske and Kinoshita, Bruno P. and Soiland-Reyes, Stian},
doi = {10.1371/journal.pone.0309210},
journal = {PLoS ONE},
keywords = {streamflow, icsc, across, eupex},
month = sep,
number = {9},
pages = {1--35},
publisher = {Public Library of Science},
title = {Recording provenance of workflow runs with RO-Crate},
url = {https://iris.unito.it/retrieve/57752f7b-9f8f-4013-8cef-5b498703d882/journal.pone.0309210.pdf},
volume = {19},
year = {2024},
bdsk-url-1 = {https://iris.unito.it/retrieve/57752f7b-9f8f-4013-8cef-5b498703d882/journal.pone.0309210.pdf},
bdsk-url-2 = {https://doi.org/10.1371/journal.pone.0309210}} A Fault Tolerance mechanism for Hybrid Scientific Workflows
Alberto Mulone, Doriana Medić, Marco Aldinucci
1st workshop about High-Performance e-Science (HiPES)
eupex icsc streamflow
In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.
@inproceedings{24:madrid:hipes,
abstract = {In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.},
address = {Madrid, Spain},
annote = {accepted},
author = {Alberto Mulone and Doriana Medi\'{c} and Marco Aldinucci},
booktitle = {1st workshop about {H}igh-{P}erformance e-{S}cience (HiPES)},
doi = {10.1007/978-3-031-90203-1_13},
editor = {Caino-Lores, Silvina and Zeinalipour, Demetris and Doudali, Thaleia Dimitra and Singh, David E. and Garz{\'o}n, Gracia Ester Mart{\'\i}n and Sousa, Leonel and Andrade, Diego and Cucinotta, Tommaso and D'Ambrosio, Donato and Diehl, Patrick and Dolz, Manuel F. and Jukan, Admela and Montella, Raffaele and Nardelli, Matteo and Garcia-Gasulla, Marta and Neuwirth, Sarah},
isbn = {978-3-031-90203-1},
keywords = {streamflow, icsc, eupex},
month = aug,
pages = {141--153},
publisher = {Springer Nature Switzerland},
title = {A Fault Tolerance mechanism for Hybrid Scientific Workflows},
volume = {15386},
year = {2024},
bdsk-url-1 = {https://doi.org/10.1007/978-3-031-90203-1_13}} Cross-Facility Federated Learning
Iacopo Colonnelli, Robert Birke, Giulio Malenza, Gianluca Mittone, Alberto Mulone, Jeroen Galjaard, Lydia Y. Chen, Sanzio Bassini, Gabriella Scipione, Jan Martinovič, Vit Vondrák, Marco Aldinucci
Procedia Computer Science
fl hpc icsc space streamflow
In a decade, AI frontier research transitioned from the researcher's workstation to thousands of high-end hardware-accelerated compute nodes. This rapid evolution shows no signs of slowing down in the foreseeable future. While top cloud providers may be able to keep pace with this growth rate, obtaining and efficiently exploiting computing resources at that scale is a daunting challenge for universities and SMEs. This work introduces the Cross-Facility Federated Learning (XFFL) framework to bridge this compute divide, extending the opportunity to efficiently exploit multiple independent data centres for extreme-scale deep learning tasks to data scientists and domain experts. XFFL relies on hybrid workflow abstractions to decouple tasks from environment-specific technicalities, reducing complexity and enhancing reusability. In addition, Federated Learning (FL) algorithms eliminate the need to move large amounts of data between different facilities, reducing time-to-solution and preserving data privacy. The XFFL approach is empirically evaluated by training a full LLaMAv2 7B instance on two facilities of the EuroHPC JU, showing how the increased computing power completely compensates for the additional overhead introduced by two data centres.
@article{24:eurohpc:xffl,
abstract = {In a decade, AI frontier research transitioned from the researcher's workstation to thousands of high-end hardware-accelerated compute nodes. This rapid evolution shows no signs of slowing down in the foreseeable future. While top cloud providers may be able to keep pace with this growth rate, obtaining and efficiently exploiting computing resources at that scale is a daunting challenge for universities and SMEs. This work introduces the Cross-Facility Federated Learning (XFFL) framework to bridge this compute divide, extending the opportunity to efficiently exploit multiple independent data centres for extreme-scale deep learning tasks to data scientists and domain experts. XFFL relies on hybrid workflow abstractions to decouple tasks from environment-specific technicalities, reducing complexity and enhancing reusability. In addition, Federated Learning (FL) algorithms eliminate the need to move large amounts of data between different facilities, reducing time-to-solution and preserving data privacy. The XFFL approach is empirically evaluated by training a full LLaMAv2 7B instance on two facilities of the EuroHPC JU, showing how the increased computing power completely compensates for the additional overhead introduced by two data centres.},
address = {Bruxelles, Belgium},
author = {Colonnelli, Iacopo and Birke, Robert and Malenza, Giulio and Mittone, Gianluca and Mulone, Alberto and Galjaard, Jeroen and Chen, Lydia Y. and Bassini, Sanzio and Scipione, Gabriella and Martinovi\v{c}, Jan and Vondr\'{a}k, Vit and Aldinucci, Marco},
booktitle = {Proceedings of the First EuroHPC user day},
date-modified = {2025-09-23 17:53:31 +0200},
doi = {10.1016/j.procs.2024.07.003},
issn = {1877-0509},
journal = {Procedia Computer Science},
keywords = {streamflow, space, icsc, fl, hpc},
pages = {3--12},
publisher = {Elsevier},
title = {Cross-Facility Federated Learning},
url = {https://www.sciencedirect.com/science/article/pii/S1877050924016909},
volume = {240},
year = {2024},
bdsk-url-1 = {https://www.sciencedirect.com/science/article/pii/S1877050924016909},
bdsk-url-2 = {https://doi.org/10.1016/j.procs.2024.07.003}} 2023
2023
Workflow Models for Heterogeneous Distributed Systems
Iacopo Colonnelli
Proceedings of the 2nd Italian Conference on Big Data and Data Science (ITADATA 2023), Naples, Italy, September 11-13, 2023
across eupex icsc jupyter-workflow streamflow
This article introduces a novel hybrid workflow abstraction that injects topology awareness directly into the definition of a distributed workflow model. In particular, the article briefly discusses the advantages brought by this approach to the design and orchestration of large-scale data-oriented workflows, the current level of support from state-of-the-art workflow systems, and some future research directions.
@inproceedings{23:colonnelli:itadata,
abstract = {This article introduces a novel hybrid workflow abstraction that injects topology awareness directly into the definition of a distributed workflow model. In particular, the article briefly discusses the advantages brought by this approach to the design and orchestration of large-scale data-oriented workflows, the current level of support from state-of-the-art workflow systems, and some future research directions.},
author = {Iacopo Colonnelli},
booktitle = {Proceedings of the 2nd Italian Conference on Big Data and Data Science ({ITADATA} 2023), Naples, Italy, September 11-13, 2023},
editor = {Bena, Nicola and Di Martino, Beniamino and Maratea, Antonio and Sperduti, Alessandro and Di Nardo, Emanuel and Ciaramella, Angelo and Montella, Raffaele and Ardagna, Claudio A.},
keywords = {across, eupex, icsc, streamflow, jupyter-workflow},
publisher = {CEUR-WS.org},
series = {{CEUR} Workshop Proceedings},
title = {Workflow Models for Heterogeneous Distributed Systems},
url = {https://ceur-ws.org/Vol-3606/invited77.pdf},
volume = {3606},
year = {2023},
bdsk-url-1 = {https://ceur-ws.org/Vol-3606/invited77.pdf}} A Systematic Mapping Study of Italian Research on Workflows
Marco Aldinucci, Elena Maria Baralis, Valeria Cardellini, Iacopo Colonnelli, Marco Danelutto, Sergio Decherchi, Giuseppe Di Modica, Luca Ferrucci, Marco Gribaudo, Francesco Iannone, Marco Lapegna, Doriana Medić, Giuseppa Muscianisi, Francesca Righetti, Eva Sciacca, Nicola Tonellotto, Mauro Tortonesi, Paolo Trunfio, Tullio Vardanega
Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023
icsc jupyter-workflow streamflow
An entire ecosystem of methodologies and tools revolves around scientific workflow management. They cover crucial non-functional requirements that standard workflow models fail to target, such as interactive execution, energy efficiency, performance portability, Big Data management, and intelligent orchestration in the Computing Continuum. Characterizing and monitoring this ecosystem is crucial to develop an informed view of current and future research directions. This work conducts a systematic mapping study of the Italian workflow research community, collecting and analyzing 25 tools and 10 applications from several scientific domains in the context of the “National Research Centre for HPC, Big Data, and Quantum Computing” (ICSC). The study aims to outline the main current research directions and determine how they address the critical needs of modern scientific applications. The findings highlight a variegated research ecosystem of tools, with a prominent interest in advanced workflow orchestration and still immature but promising efforts toward energy efficiency.
@inproceedings{WORKS2023,
abstract = {An entire ecosystem of methodologies and tools revolves around scientific workflow management. They cover crucial non-functional requirements that standard workflow models fail to target, such as interactive execution, energy efficiency, performance portability, Big Data management, and intelligent orchestration in the Computing Continuum. Characterizing and monitoring this ecosystem is crucial to develop an informed view of current and future research directions. This work conducts a systematic mapping study of the Italian workflow research community, collecting and analyzing 25 tools and 10 applications from several scientific domains in the context of the ``National Research Centre for HPC, Big Data, and Quantum Computing'' (ICSC). The study aims to outline the main current research directions and determine how they address the critical needs of modern scientific applications. The findings highlight a variegated research ecosystem of tools, with a prominent interest in advanced workflow orchestration and still immature but promising efforts toward energy efficiency.},
address = {Denver, CO, USA},
author = {Aldinucci, Marco and Maria Baralis, Elena and Cardellini, Valeria and Colonnelli, Iacopo and Danelutto, Marco and Decherchi, Sergio and Di Modica, Giuseppe and Ferrucci, Luca and Gribaudo, Marco and Iannone, Francesco and Lapegna, Marco and Medi\'{c}, Doriana and Muscianisi, Giuseppa and Righetti, Francesca and Sciacca, Eva and Tonellotto, Nicola and Tortonesi, Mauro and Trunfio, Paolo and Vardanega, Tullio},
booktitle = {Proceedings of the {SC} '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, {SC-W} 2023},
doi = {10.1145/3624062.3624285},
keywords = {streamflow, jupyter-workflow, icsc},
month = nov,
pages = {2065--2076},
publisher = {{ACM}},
title = {A Systematic Mapping Study of Italian Research on Workflows},
url = {https://doi.org/10.1145/3624062.3624285},
year = {2023},
bdsk-url-1 = {https://doi.org/10.1145/3624062.3624285}} Porting the Variant Calling Pipeline for NGS data in cloud-HPC environment
Alberto Mulone, Sherine Awad, Davide Chiarugi, Marco Aldinucci
47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023
across icsc streamflow
In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.
@inproceedings{23:mulone:wide:vcp,
abstract = {In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.},
address = {Torino, Italy},
author = {Alberto Mulone and Sherine Awad and Davide Chiarugi and Marco Aldinucci},
booktitle = {47th {IEEE} Annual Computers, Software, and Applications Conference, {COMPSAC} 2023},
doi = {10.1109/COMPSAC57700.2023.00288},
editor = {Hossain Shahriar and Yuuichi Teranishi and Alfredo Cuzzocrea and Moushumi Sharmin and Dave Towey and A. K. M. Jahangir Alam Majumder and Hiroki Kashiwazaki and Ji{-}Jiang Yang and Michiharu Takemoto and Nazmus Sakib and Ryohei Banno and Sheikh Iqbal Ahamed},
keywords = {across, icsc, streamflow},
pages = {1858--1863},
publisher = {{IEEE}},
title = {Porting the {V}ariant {C}alling {P}ipeline for {NGS} data in cloud-{HPC} environment},
url = {https://iris.unito.it/bitstream/2318/1919364/1/paper.pdf},
year = {2023},
bdsk-url-1 = {https://iris.unito.it/bitstream/2318/1919364/1/paper.pdf},
bdsk-url-2 = {https://doi.org/10.1109/COMPSAC57700.2023.00288}} Federated Learning meets HPC and cloud
Iacopo Colonnelli, Bruno Casella, Gianluca Mittone, Yasir Arfat, Barbara Cantalupo, Roberto Esposito, Alberto R. Martinelli, Doriana Medić, Marco Aldinucci
Astrophysics and Space Science Proceedings
across eupilot fl streamflow
HPC and AI are fated to meet for several reasons. This article will discuss some of them and argue why this will happen through the set of methods and technologies that underpin cloud computing. As a paradigmatic example, we present a new federated learning system that collaboratively trains a deep learning model in different supercomputing centers. The system is based on the StreamFlow workflow manager designed for hybrid cloud-HPC infrastructures.
@inproceedings{22:ml4astro,
abstract = {HPC and AI are fated to meet for several reasons. This article will discuss some of them and argue why this will happen through the set of methods and technologies that underpin cloud computing. As a paradigmatic example, we present a new federated learning system that collaboratively trains a deep learning model in different supercomputing centers. The system is based on the StreamFlow workflow manager designed for hybrid cloud-HPC infrastructures.},
address = {Catania, Italy},
author = {Colonnelli, Iacopo and Casella, Bruno and Mittone, Gianluca and Arfat, Yasir and Cantalupo, Barbara and Esposito, Roberto and Martinelli, Alberto R. and Medi\'{c}, Doriana and Aldinucci, Marco},
booktitle = {Astrophysics and Space Science Proceedings},
date-modified = {2024-04-01 11:12:57 +0200},
doi = {10.1007/978-3-031-34167-0_39},
editor = {Bufano, Filomena and Riggi, Simone and Sciacca, Eva and Schillir\`{o}, Francesco},
isbn = {978-3-031-34167-0},
keywords = {across, eupilot, streamflow, fl},
pages = {193--199},
publisher = {Springer},
title = {Federated Learning meets {HPC} and cloud},
url = {https://iris.unito.it/retrieve/3ac66baa-9d9a-4e9f-94a5-13700694d8aa/ML4Astro.pdf},
volume = {60},
year = {2023},
bdsk-url-1 = {https://iris.unito.it/retrieve/3ac66baa-9d9a-4e9f-94a5-13700694d8aa/ML4Astro.pdf}} Bringing Cell Subpopulation Discovery on a Cloud-HPC Using rCASC and StreamFlow
Sandro Gepiro Contaldo, Luca Alessandri, Iacopo Colonnelli, Marco Beccuti, Marco Aldinucci
Single Cell Transcriptomics: Methods and Protocols
streamflow
The idea behind novel single-cell RNA sequencing (scRNA-seq) pipelines is to isolate single cells through microfluidic approaches and generate sequencing libraries in which the transcripts are tagged to track their cell of origin. Modern scRNA-seq platforms are capable of analyzing up to many thousands of cells in each run. Then, combined with massive high-throughput sequencing producing billions of reads, scRNA-seq allows the assessment of fundamental biological properties of cell populations and biological systems at unprecedented resolution.
@inbook{Contaldo2023,
abstract = {The idea behind novel single-cell RNA sequencing (scRNA-seq) pipelines is to isolate single cells through microfluidic approaches and generate sequencing libraries in which the transcripts are tagged to track their cell of origin. Modern scRNA-seq platforms are capable of analyzing up to many thousands of cells in each run. Then, combined with massive high-throughput sequencing producing billions of reads, scRNA-seq allows the assessment of fundamental biological properties of cell populations and biological systems at unprecedented resolution.},
address = {New York, NY},
author = {Contaldo, Sandro Gepiro and Alessandri, Luca and Colonnelli, Iacopo and Beccuti, Marco and Aldinucci, Marco},
booktitle = {Single Cell Transcriptomics: Methods and Protocols},
doi = {10.1007/978-1-0716-2756-3_17},
editor = {Calogero, Raffaele Adolfo and Benes, Vladimir},
isbn = {978-1-0716-2756-3},
keywords = {streamflow},
pages = {337--345},
publisher = {Springer {US}},
title = {Bringing Cell Subpopulation Discovery on a Cloud-{HPC} Using {rCASC} and {StreamFlow}},
url = {https://datacloud.di.unito.it/index.php/s/KMfKo4m7GTGdZmF},
year = {2023},
bdsk-url-1 = {https://doi.org/10.1007/978-1-0716-2756-3_17}} 2022
2022
The DeepHealth Toolkit: A Key European Free and Open-Source Software for Deep Learning and Computer Vision Ready to Exploit Heterogeneous HPC and Cloud Architectures
Marco Aldinucci, David Atienza, Federico Bolelli, Mónica Caballero, Iacopo Colonnelli, José Flich, Jon Ander Gómez, David González, Costantino Grana, Marco Grangetto, Simone Leo, Pedro López, Dana Oniga, Roberto Paredes, Luca Pireddu, Eduardo Quiñones, Tatiana Silva, Enzo Tartaglione, Marina Zapater
Technologies and Applications for Big Data Value
deephealth streamflow
At the present time, we are immersed in the convergence between Big Data, High-Performance Computing and Artificial Intelligence. Technological progress in these three areas has accelerated in recent years, forcing different players like software companies and stakeholders to move quickly. The European Union is dedicating a lot of resources to maintain its relevant position in this scenario, funding projects to implement large-scale pilot testbeds that combine the latest advances in Artificial Intelligence, High-Performance Computing, Cloud and Big Data technologies. The DeepHealth project is an example focused on the health sector whose main outcome is the DeepHealth toolkit, a European unified framework that offers deep learning and computer vision capabilities, completely adapted to exploit underlying heterogeneous High-Performance Computing, Big Data and cloud architectures, and ready to be integrated into any software platform to facilitate the development and deployment of new applications for specific problems in any sector. This toolkit is intended to be one of the European contributions to the field of AI. This chapter introduces the toolkit with its main components and complementary tools, providing a clear view to facilitate and encourage its adoption and wide use by the European community of developers of AI-based solutions and data scientists working in the healthcare sector and others.
@incollection{22:TABDV,
abstract = {At the present time, we are immersed in the convergence between Big Data, High-Performance Computing and Artificial Intelligence. Technological progress in these three areas has accelerated in recent years, forcing different players like software companies and stakeholders to move quickly. The European Union is dedicating a lot of resources to maintain its relevant position in this scenario, funding projects to implement large-scale pilot testbeds that combine the latest advances in Artificial Intelligence, High-Performance Computing, Cloud and Big Data technologies. The DeepHealth project is an example focused on the health sector whose main outcome is the DeepHealth toolkit, a European unified framework that offers deep learning and computer vision capabilities, completely adapted to exploit underlying heterogeneous High-Performance Computing, Big Data and cloud architectures, and ready to be integrated into any software platform to facilitate the development and deployment of new applications for specific problems in any sector. This toolkit is intended to be one of the European contributions to the field of AI. This chapter introduces the toolkit with its main components and complementary tools, providing a clear view to facilitate and encourage its adoption and wide use by the European community of developers of AI-based solutions and data scientists working in the healthcare sector and others.},
address = {Cham},
author = {Aldinucci, Marco and Atienza, David and Bolelli, Federico and Caballero, M\'{o}nica and Colonnelli, Iacopo and Flich, Jos\'{e} and Ander G\'{o}mez, Jon and Gonz\'{a}lez, David and Grana, Costantino and Grangetto, Marco and Leo, Simone and L\'{o}pez, Pedro and Oniga, Dana and Paredes, Roberto and Pireddu, Luca and Qui\~{n}ones, Eduardo and Silva, Tatiana and Tartaglione, Enzo and Zapater, Marina},
booktitle = {Technologies and Applications for Big Data Value},
chapter = {9},
doi = {10.1007/978-3-030-78307-5_9},
editor = {Edward Curry and S\"{o}ren Auer and Arne J. Berre and Andreas Metzger and Maria S. Perez and Sonja Zillner},
isbn = {978-3-030-78307-5},
keywords = {deephealth, streamflow},
pages = {183--202},
publisher = {Springer International Publishing},
title = {The {DeepHealth} Toolkit: A Key European Free and Open-Source Software for Deep Learning and Computer Vision Ready to Exploit Heterogeneous {HPC} and {C}loud Architectures},
url = {https://link.springer.com/content/pdf/10.1007/978-3-030-78307-5_9.pdf},
year = {2022},
bdsk-url-1 = {https://link.springer.com/content/pdf/10.1007/978-3-030-78307-5_9.pdf},
bdsk-url-2 = {https://doi.org/10.1007/978-3-030-78307-5_9}} The DeepHealth HPC Infrastructure: Leveraging Heterogenous HPC and Cloud Computing Infrastructures for IA-based Medical Solutions
Eduardo Quiñones, Jesus Perales, Jorge Ejarque, Asaf Badouh, Santiago Marco, Fabrice Auzanneau, François Galea, David González, José Ramón Hervás, Tatiana Silva, Iacopo Colonnelli, Barbara Cantalupo, Marco Aldinucci, Enzo Tartaglione, Rafael Tornero, José Flich, Jose Maria Martinez, David Rodriguez, Izan Catalán, Jorge Garcia, Carles Hernández
HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision
deephealth streamflow
This chapter presents the DeepHealth HPC toolkit for an efficient execution of deep learning (DL) medical application into HPC and cloud-computing infrastructures, featuring many-core, GPU, and FPGA acceleration devices. The toolkit offers to the European Computer Vision Library and the European Distributed Deep Learning Library (EDDL), developed in the DeepHealth project as well, the mechanisms to distribute and parallelize DL operations on HPC and cloud infrastructures in a fully transparent way. The toolkit implements workflow managers used to orchestrate HPC workloads for an efficient parallelization of EDDL training operations on HPC and cloud infrastructures, and includes the parallel programming models for an efficient execution EDDL inference and training operations on many-core, GPUs and FPGAs acceleration devices.
@incollection{22:deephealth:HPCbook,
abstract = {This chapter presents the DeepHealth HPC toolkit for an efficient execution of deep learning (DL) medical application into HPC and cloud-computing infrastructures, featuring many-core, GPU, and FPGA acceleration devices. The toolkit offers to the European Computer Vision Library and the European Distributed Deep Learning Library (EDDL), developed in the DeepHealth project as well, the mechanisms to distribute and parallelize DL operations on HPC and cloud infrastructures in a fully transparent way. The toolkit implements workflow managers used to orchestrate HPC workloads for an efficient parallelization of EDDL training operations on HPC and cloud infrastructures, and includes the parallel programming models for an efficient execution EDDL inference and training operations on many-core, GPUs and FPGAs acceleration devices.},
address = {Boca Raton, Florida},
author = {Qui\~{n}ones, Eduardo and Perales, Jesus and Ejarque, Jorge and Badouh, Asaf and Marco, Santiago and Auzanneau, Fabrice and Galea, Fran\c{c}ois and Gonz\'{a}lez, David and Herv\'{a}s, Jos\'{e} Ram\'{o}n and Silva, Tatiana and Colonnelli, Iacopo and Cantalupo, Barbara and Aldinucci, Marco and Tartaglione, Enzo and Tornero, Rafael and Flich, Jos\'{e} and Martinez, Jose Maria and Rodriguez, David and Catal\'{a}n, Izan and Garcia, Jorge and Hern\'{a}ndez, Carles},
booktitle = {{HPC}, Big Data, and {AI} Convergence Towards Exascale: Challenge and Vision},
chapter = {10},
doi = {10.1201/9781003176664},
editor = {Olivier Terzo and Jan Martinovi\v{c}},
isbn = {978-1-0320-0984-1},
keywords = {deephealth, streamflow},
pages = {191--216},
publisher = {{CRC} Press},
title = {The {DeepHealth} {HPC} Infrastructure: Leveraging Heterogenous {HPC} and Cloud Computing Infrastructures for {IA}-based Medical Solutions},
url = {https://iris.unito.it/retrieve/handle/2318/1832050/912413/Preprint.pdf},
year = {2022},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1832050/912413/Preprint.pdf},
bdsk-url-2 = {https://doi.org/10.1201/9781003176664}} Distributed workflows with Jupyter
Iacopo Colonnelli, Marco Aldinucci, Barbara Cantalupo, Luca Padovani, Sergio Rabellino, Concetto Spampinato, Roberto Morelli, Rosario Di Carlo, Nicol`o Magini, Carlo Cavazzoni
Future Generation Computer Systems
across deephealth jupyter-workflow streamflow
The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code's business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks' potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.
@article{21:FGCS:jupyflow,
abstract = {The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code's business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks' potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.},
author = {Colonnelli, Iacopo and Aldinucci, Marco and Cantalupo, Barbara and Padovani, Luca and Rabellino, Sergio and Spampinato, Concetto and Morelli, Roberto and Di Carlo, Rosario and Magini, Nicol{`o} and Cavazzoni, Carlo},
doi = {10.1016/j.future.2021.10.007},
issn = {0167-739X},
journal = {Future Generation Computer Systems},
keywords = {streamflow, jupyter-workflow, deephealth, across},
pages = {282--298},
title = {Distributed workflows with {Jupyter}},
url = {https://www.sciencedirect.com/science/article/pii/S0167739X21003976},
volume = {128},
year = {2022},
bdsk-url-1 = {https://www.sciencedirect.com/science/article/pii/S0167739X21003976},
bdsk-url-2 = {https://doi.org/10.1016/j.future.2021.10.007}} 2021
2021
TEXTAROSSA: Towards EXtreme scale Technologies and Accelerators for euROhpc hw/Sw Supercomputing Applications for exascale
Giovanni Agosta, William Fornaciari, Andrea Galimberti, Giuseppe Massari, Federico Reghenzani, Federico Terraneo, Davide Zoni, Carlo Brandolese, Massimo Celino, Francesco Iannone, Paolo Palazzari, Giuseppe Zummo, Massimo Bernaschi, Pasqua D'Ambra, Sergio Saponara, Marco Danelutto, Massimo Torquati, Marco Aldinucci, Yasir Arfat, Barbara Cantalupo, Iacopo Colonnelli, Roberto Esposito, Alberto R. Martinelli, Gianluca Mittone, Olivier Beaumont, Berenger Bramas, Lionel Eyraud-Dubois, Brice Goglin, Abdou Guermouche, Raymond Namyst, Samuel Thibault, Antonio Filgueras, Miquel Vidal, Carlos Alvarez, Xavier Martorell, Ariel Oleksiak, Michal Kulczewski, Alessandro Lonardo, Piero Vicini, Francesco Lo Cicero, Francesco Simula, Andrea Biagioni, Paolo Cretaro, Ottorino Frezza, Pier Stanislao Paolucci, Matteo Turisini, Francesco Giacomini, Tommaso Boccali, Simone Montangero, Roberto Ammendola
Proc. of the 24th Euromicro Conference on Digital System Design (DSD)
hpc streamflow textarossa
To achieve high performance and high energy efficiency on near-future exascale computing systems, three key technology gaps needs to be bridged. These gaps include: energy efficiency and thermal control; extreme computation efficiency via HW acceleration and new arithmetics; methods and tools for seamless integration of reconfigurable accelerators in heterogeneous HPC multi-node platforms. TEXTAROSSA aims at tackling this gap through a co-design approach to heterogeneous HPC solutions, supported by the integration and extension of HW and SW IPs, programming models and tools derived from European research.
@inproceedings{21:DSD:textarossa,
abstract = {To achieve high performance and high energy efficiency on near-future exascale computing systems, three key technology gaps needs to be bridged. These gaps include: energy efficiency and thermal control; extreme computation efficiency via HW acceleration and new arithmetics; methods and tools for seamless integration of reconfigurable accelerators in heterogeneous HPC multi-node platforms. TEXTAROSSA aims at tackling this gap through a co-design approach to heterogeneous HPC solutions, supported by the integration and extension of HW and SW IPs, programming models and tools derived from European research.},
address = {Palermo, Italy},
author = {Agosta, Giovanni and Fornaciari, William and Galimberti, Andrea and Massari, Giuseppe and Reghenzani, Federico and Terraneo, Federico and Zoni, Davide and Brandolese, Carlo and Celino, Massimo and Iannone, Francesco and Palazzari, Paolo and Zummo, Giuseppe and Bernaschi, Massimo and D'Ambra, Pasqua and Saponara, Sergio and Danelutto, Marco and Torquati, Massimo and Aldinucci, Marco and Arfat, Yasir and Cantalupo, Barbara and Colonnelli, Iacopo and Esposito, Roberto and Martinelli, Alberto R. and Mittone, Gianluca and Beaumont, Olivier and Bramas, Berenger and Eyraud-Dubois, Lionel and Goglin, Brice and Guermouche, Abdou and Namyst, Raymond and Thibault, Samuel and Filgueras, Antonio and Vidal, Miquel and Alvarez, Carlos and Martorell, Xavier and Oleksiak, Ariel and Kulczewski, Michal and Lonardo, Alessandro and Vicini, Piero and Lo Cicero, Francesco and Simula, Francesco and Biagioni, Andrea and Cretaro, Paolo and Frezza, Ottorino and Stanislao Paolucci, Pier and Turisini, Matteo and Giacomini, Francesco and Boccali, Tommaso and Montangero, Simone and Ammendola, Roberto},
booktitle = {Proc. of the 24th Euromicro Conference on Digital System Design ({DSD})},
date-added = {2021-09-04 12:07:42 +0200},
date-modified = {2021-09-04 12:23:41 +0200},
doi = {10.1109/DSD53832.2021.00051},
keywords = {textarossa, streamflow, hpc},
month = aug,
publisher = {IEEE},
title = {{TEXTAROSSA}: Towards EXtreme scale Technologies and Accelerators for euROhpc hw/Sw Supercomputing Applications for exascale},
year = {2021},
bdsk-url-1 = {https://doi.org/10.1109/DSD53832.2021.00051}} Bringing AI pipelines onto cloud-HPC: setting a baseline for accuracy of COVID-19 diagnosis
Iacopo Colonnelli, Barbara Cantalupo, Concetto Spampinato, Matteo Pennisi, Marco Aldinucci
ENEA CRESCO in the fight against COVID-19
streamflow
HPC is an enabling platform for AI. The introduction of AI workloads in the HPC applications basket has non-trivial consequences both on the way of designing AI applications and on the way of providing HPC computing. This is the leitmotif of the convergence between HPC and AI. The formalized definition of AI pipelines is one of the milestones of HPC-AI convergence. If well conducted, it allows, on the one hand, to obtain portable and scalable applications. On the other hand, it is crucial for the reproducibility of scientific pipelines. In this work, we advocate the StreamFlow Workflow Management System as a crucial ingredient to define a parametric pipeline, called “CLAIRE COVID-19 Universal Pipeline”, which is able to explore the optimization space of methods to classify COVID-19 lung lesions from CT scans, compare them for accuracy, and therefore set a performance baseline. The universal pipeline automatizes the training of many different Deep Neural Networks (DNNs) and many different hyperparameters. It, therefore, requires a massive computing power, which is found in traditional HPC infrastructure thanks to the portability-by-design of pipelines designed with StreamFlow. Using the universal pipeline, we identified a DNN reaching over 90% accuracy in detecting COVID-19 lesions in CT scans.
@inproceedings{21:covi:enea,
abstract = {HPC is an enabling platform for AI. The introduction of AI workloads in the HPC applications basket has non-trivial consequences both on the way of designing AI applications and on the way of providing HPC computing. This is the leitmotif of the convergence between HPC and AI. The formalized definition of AI pipelines is one of the milestones of HPC-AI convergence. If well conducted, it allows, on the one hand, to obtain portable and scalable applications. On the other hand, it is crucial for the reproducibility of scientific pipelines. In this work, we advocate the StreamFlow Workflow Management System as a crucial ingredient to define a parametric pipeline, called ``CLAIRE COVID-19 Universal Pipeline'', which is able to explore the optimization space of methods to classify COVID-19 lung lesions from CT scans, compare them for accuracy, and therefore set a performance baseline. The universal pipeline automatizes the training of many different Deep Neural Networks (DNNs) and many different hyperparameters. It, therefore, requires a massive computing power, which is found in traditional HPC infrastructure thanks to the portability-by-design of pipelines designed with StreamFlow. Using the universal pipeline, we identified a DNN reaching over 90\% accuracy in detecting COVID-19 lesions in CT scans.},
author = {Colonnelli, Iacopo and Cantalupo, Barbara and Spampinato, Concetto and Pennisi, Matteo and Aldinucci, Marco},
booktitle = {ENEA CRESCO in the fight against COVID-19},
doi = {10.5281/zenodo.5151511},
editor = {Francesco Iannone},
keywords = {streamflow},
publisher = {ENEA},
title = {Bringing AI pipelines onto cloud-HPC: setting a baseline for accuracy of COVID-19 diagnosis},
url = {https://iris.unito.it/retrieve/handle/2318/1796029/779853/21_AI-pipelines_ENEA-COVID19.pdf},
year = {2021},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1796029/779853/21_AI-pipelines_ENEA-COVID19.pdf},
bdsk-url-2 = {https://doi.org/10.5281/zenodo.5151511}} HPC Application Cloudification: The StreamFlow Toolkit
Iacopo Colonnelli, Barbara Cantalupo, Roberto Esposito, Matteo Pennisi, Concetto Spampinato, Marco Aldinucci
12th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures and 10th Workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms (PARMA-DITAM 2021)
deephealth hpc4ai streamflow system-sw
Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud infrastructure can be offloaded to HPC environments to speed them up. We introduce StreamFlow, a novel Workflow Management System that supports such a design pattern and makes it possible to run the steps of a standard workflow model on independent processing elements with no shared storage. We validated the proposed approach's effectiveness on the CLAIRE COVID-19 universal pipeline, i.e. a reproducible workflow capable of automating the comparison of (possibly all) state-of-the-art pipelines for the diagnosis of COVID-19 interstitial pneumonia from CT scans images based on Deep Neural Networks (DNNs).
@inproceedings{colonnelli_et_al:OASIcs.PARMA-DITAM.2021.5,
abstract = {Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud infrastructure can be offloaded to HPC environments to speed them up. We introduce StreamFlow, a novel Workflow Management System that supports such a design pattern and makes it possible to run the steps of a standard workflow model on independent processing elements with no shared storage. We validated the proposed approach's effectiveness on the CLAIRE COVID-19 universal pipeline, i.e. a reproducible workflow capable of automating the comparison of (possibly all) state-of-the-art pipelines for the diagnosis of COVID-19 interstitial pneumonia from CT scans images based on Deep Neural Networks (DNNs).},
address = {Dagstuhl, Germany},
annote = {Projects: cloud computing, distributed computing, high-performance computing, streamflow, workflow management systems},
author = {Colonnelli, Iacopo and Cantalupo, Barbara and Esposito, Roberto and Pennisi, Matteo and Spampinato, Concetto and Aldinucci, Marco},
booktitle = {12th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures and 10th Workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms (PARMA-DITAM 2021)},
date-modified = {2025-05-24 09:12:41 +0200},
doi = {10.4230/OASIcs.PARMA-DITAM.2021.5},
editor = {Bispo, Jo\~{a}o and Cherubin, Stefano and Flich, Jos\'{e}},
isbn = {978-3-95977-181-8},
issn = {2190-6807},
keywords = {system-sw, deephealth, hpc4ai, streamflow},
pages = {5:1--5:13},
publisher = {Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
series = {Open Access Series in Informatics (OASIcs)},
title = {{HPC Application Cloudification: The StreamFlow Toolkit}},
url = {https://drops.dagstuhl.de/opus/volltexte/2021/13641/pdf/OASIcs-PARMA-DITAM-2021-5.pdf},
urn = {urn:nbn:de:0030-drops-136419},
volume = {88},
year = {2021},
bdsk-url-1 = {https://drops.dagstuhl.de/opus/volltexte/2021/13641/pdf/OASIcs-PARMA-DITAM-2021-5.pdf},
bdsk-url-2 = {https://doi.org/10.4230/OASIcs.PARMA-DITAM.2021.5}} StreamFlow: cross-breeding cloud with HPC
Iacopo Colonnelli, Barbara Cantalupo, Ivan Merelli, Marco Aldinucci
IEEE Transactions on Emerging Topics in Computing
deephealth hpc4ai streamflow
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single cell transcriptomic data analysis workflow.
@article{20Lstreamflow:tetc,
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single cell transcriptomic data analysis workflow.},
author = {Iacopo Colonnelli and Barbara Cantalupo and Ivan Merelli and Marco Aldinucci},
date-added = {2020-08-27 09:29:49 +0200},
date-modified = {2020-08-27 09:36:33 +0200},
doi = {10.1109/TETC.2020.3019202},
journal = {{IEEE} {T}ransactions on {E}merging {T}opics in {C}omputing},
keywords = {deephealth, hpc4ai, streamflow},
number = {4},
pages = {1723--1737},
title = {{StreamFlow}: cross-breeding cloud with {HPC}},
url = {https://arxiv.org/pdf/2002.01558},
volume = {9},
year = {2021},
bdsk-url-1 = {https://arxiv.org/pdf/2002.01558},
bdsk-url-2 = {https://doi.org/10.1109/TETC.2020.3019202}} 2025
2025
Hybrid Workflows in the Heterogeneous Era
Iacopo Colonnelli
capio eupex streamflow
Thanks to their generality, workflow models represent a powerful abstraction for designing complex applications and executing them on large-scale distributed architectures. However, several additional challenges appear when transitioning from cloud/HPC environments to the entire computing continuum. Continuum execution environments are fully distributed and modular, and modules can be heterogeneous and independent of each other. In addition, continuum workflows often rely on multiple intercommunicating agents that form complex micro-services architectures. Different agents deal with different communication and parallelization paradigms: network-based stream processing at the edge and file-based batch processing on HPC facilities. Finally, support for efficient interactive workflows in the continuum remains an open research problem. This webinar delves into these challenges, exploring how current research addresses them with innovatives techniques and tools (i.e., hybrid workflows and StreamFlow, literate workflows and Jupyter Workflow, transparent streaming workflows and CAPIO), and suggesting potential directions for future investigation.
@misc{25:eupex:streamflow,
abstract = {Thanks to their generality, workflow models represent a powerful abstraction for designing complex applications and executing them on large-scale distributed architectures. However, several additional challenges appear when transitioning from cloud/HPC environments to the entire computing continuum. Continuum execution environments are fully distributed and modular, and modules can be heterogeneous and independent of each other. In addition, continuum workflows often rely on multiple intercommunicating agents that form complex micro-services architectures. Different agents deal with different communication and parallelization paradigms: network-based stream processing at the edge and file-based batch processing on HPC facilities. Finally, support for efficient interactive workflows in the continuum remains an open research problem. This webinar delves into these challenges, exploring how current research addresses them with innovatives techniques and tools (i.e., hybrid workflows and StreamFlow, literate workflows and Jupyter Workflow, transparent streaming workflows and CAPIO), and suggesting potential directions for future investigation.},
author = {Iacopo Colonnelli},
keywords = {eupex, streamflow, capio},
month = feb,
title = {Hybrid Workflows in the Heterogeneous Era},
url = {https://datacloud.di.unito.it/index.php/s/Y76mXwCRKJ2CDa3},
year = 2025,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/Y76mXwCRKJ2CDa3}} Cross-Platform Full Waveform Inversion (CPFWI)
Iacopo Colonnelli
icsc streamflow
@misc{25:icsc:cpfwi,
address = {Milano, Italy},
author = {Iacopo Colonnelli},
howpublished = {ICSC Spoke 1 Event},
keywords = {icsc, streamflow},
month = mar,
title = {Cross-Platform Full Waveform Inversion (CPFWI)},
url = {https://datacloud.di.unito.it/index.php/s/3kbBHf39RaFCA9X},
year = 2025,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/3kbBHf39RaFCA9X}} Cross-Facility Federated Learning
Gianluca Mittone, Alberto Mulone, Iacopo Colonnelli, Robert Birke, Marco Aldinucci
eupilot fl icsc space streamflow
Big-tech companies pre-train SOTA LLMs on special-purpose, private HPCs, while public research centres lack the resources to compete. We advocate a new take on large model training, e.g., LLMs, called xFFL, which leverages federated learning as an enabling technique to exploit geographically distributed computing power to bridge such digital divide. This work introduces a proof-of-concept federated training of LLaMA-3 8B on three EuroHPC Top500 facilities, proving the viability of leveraging cross-facility publicly available computational power to sustain SOTA LLM workloads.
@misc{25:cineca:xffl,
abstract = {Big-tech companies pre-train SOTA LLMs on special-purpose, private HPCs, while public research centres lack the resources to compete. We advocate a new take on large model training, e.g., LLMs, called xFFL, which leverages federated learning as an enabling technique to exploit geographically distributed computing power to bridge such digital divide. This work introduces a proof-of-concept federated training of LLaMA-3 8B on three EuroHPC Top500 facilities, proving the viability of leveraging cross-facility publicly available computational power to sustain SOTA LLM workloads.},
address = {Bologna, Italy},
author = {Mittone, Gianluca and Mulone, Alberto and Colonnelli, Iacopo and Birke, Robert and Aldinucci, Marco},
howpublished = {AI & HPC User Forums and HPC User Day},
keywords = {icsc, eupilot, space, fl, streamflow},
month = mar,
title = {Cross-Facility Federated Learning},
url = {https://datacloud.di.unito.it/index.php/s/oXoeHLytc9CZ5KR},
year = 2025,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/oXoeHLytc9CZ5KR}} 2024
2024
Cross-Platform Full Waveform Inversion
Alberto Mulone
icsc streamflow
Presentation about the Innovation Grant in collaboration with ENI
@misc{24:icsc:spoke1:eni,
abstract = {Presentation about the Innovation Grant in collaboration with ENI},
address = {Torino, Italy},
author = {Mulone, Alberto},
howpublished = {ICSC - Spoke 1 meeting},
keywords = {icsc, streamflow},
month = feb,
title = {Cross-Platform Full Waveform Inversion},
url = {https://datacloud.di.unito.it/index.php/s/M3HkxA5wsBPS5ro},
year = 2024,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/M3HkxA5wsBPS5ro}} Parallel Computing at the University of Torino
Marco Aldinucci
ai fl hpc hpc4ai icsc streamflow
@misc{24:alpha:unito,
author = {Marco Aldinucci},
howpublished = {University of Torino},
keywords = {hpc4ai, icsc, hpc, ai, fl, streamflow},
month = mar,
title = {Parallel Computing at the University of Torino},
year = 2024} CWL in the HPC Ecosystem
Iacopo Colonnelli
across eupex icsc space streamflow
@misc{24:icolonne:cwl4hpccern,
address = {CERN, Meyrin, Switzerland},
author = {Colonnelli, Iacopo},
howpublished = {Workshop on workflow languages for {HEP} analysis},
keywords = {across, eupex, space, icsc, streamflow},
month = apr,
title = {CWL in the HPC Ecosystem},
url = {https://datacloud.di.unito.it/index.php/s/PRmqdwWHt6P2PH7},
year = 2024,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/PRmqdwWHt6P2PH7}} Dynamic hybrid workflows for Deep Learning on HPC infrastructure
Iacopo Colonnelli
icsc jupyter-workflow streamflow
Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher's laptop to an entire HPC facility.
@misc{24:icolonne:ictp,
abstract = {Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher's laptop to an entire HPC facility.},
address = {Trieste, Italy},
author = {Iacopo Colonnelli},
keywords = {icsc, streamflow, jupyter-workflow},
month = may,
title = {Dynamic hybrid workflows for Deep Learning on HPC infrastructure},
url = {https://datacloud.di.unito.it/index.php/s/EaFHJEKNbW5oXeq},
year = 2024,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/EaFHJEKNbW5oXeq}} A Fault Tolerance mechanism for Hybrid Scientific Workflows
Alberto Mulone, Doriana Medić, Marco Aldinucci
eupex icsc streamflow
In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.
@misc{24:madrid:hipes:talk,
abstract = {In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.},
address = {Madrid, Spain},
author = {Alberto Mulone and Doriana Medi\'{c} and Marco Aldinucci},
howpublished = {1st workshop about {H}igh-{P}erformance e-{S}cience (HiPES)},
keywords = {streamflow, icsc, eupex},
month = aug,
title = {A Fault Tolerance mechanism for Hybrid Scientific Workflows},
url = {https://datacloud.di.unito.it/index.php/s/9Ddj6fGgmDbLDXj},
year = 2024,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/9Ddj6fGgmDbLDXj}} Enabling Cross-Facility LLMs Pre-Training
Gianluca Mittone, Alberto Mulone, Iacopo Colonnelli, Robert Birke, Marco Aldinucci
eupilot fl icsc space streamflow
Big-tech companies pre-train SOTA LLMs on special-purpose, private HPCs, while public research centres lack the resources to compete. We advocate a new take on large model training, e.g., LLMs, called xFFL, which leverages federated learning as an enabling technique to exploit geographically distributed computing power to bridge such digital divide. This work introduces a proof-of-concept federated training of LLaMA-3 8B on three EuroHPC Top500 facilities, proving the viability of leveraging cross-facility publicly available computational power to sustain SOTA LLM workloads.
@misc{24:mittone:TPC,
abstract = {Big-tech companies pre-train SOTA LLMs on special-purpose, private HPCs, while public research centres lack the resources to compete. We advocate a new take on large model training, e.g., LLMs, called xFFL, which leverages federated learning as an enabling technique to exploit geographically distributed computing power to bridge such digital divide. This work introduces a proof-of-concept federated training of LLaMA-3 8B on three EuroHPC Top500 facilities, proving the viability of leveraging cross-facility publicly available computational power to sustain SOTA LLM workloads.},
address = {Atlanta, GE, USA},
author = {Mittone, Gianluca and Mulone, Alberto and Colonnelli, Iacopo and Birke, Robert and Aldinucci, Marco},
howpublished = {Accelerating the Development and Use of Generative AI for Science and Engineering: The Trillion Parameter Consortium ({TPC})},
keywords = {icsc, eupilot, space, fl, streamflow},
month = nov,
title = {Enabling Cross-Facility LLMs Pre-Training},
url = {https://datacloud.di.unito.it/index.php/s/DRgm8ebBkKQgD2d},
year = 2024,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/DRgm8ebBkKQgD2d}} 2023
2023
Standardised Workflows at EBRAINS
Sofia Karvounari, Eleni Mathioulaki, Michael R. Crusoe, Iacopo Colonnelli
across eupex space streamflow
A hands-on training offer for Standardised Workflows in EBRAINS. A short presentation will be used as an introduction, while the main hands-on session will provide information about Writing and Executing Standardised Workflows. TC will give some guidelines, so attendees can experiment with writing CWL tools and workflows and then they will be given access to VM to execute these workflows. The Workflows Dashboard will be also presented during the same session, offering to the attendees the opportunity to understand the different functionalities, use it with TC support and provide useful comments.
@misc{23:HBPSummit,
abstract = {A hands-on training offer for Standardised Workflows in EBRAINS. A short presentation will be used as an introduction, while the main hands-on session will provide information about Writing and Executing Standardised Workflows. TC will give some guidelines, so attendees can experiment with writing CWL tools and workflows and then they will be given access to VM to execute these workflows. The Workflows Dashboard will be also presented during the same session, offering to the attendees the opportunity to understand the different functionalities, use it with TC support and provide useful comments.},
address = {Marseille, France},
annote = {invited},
author = {Sofia Karvounari and Eleni Mathioulaki and Michael R. Crusoe and Iacopo Colonnelli},
howpublished = {Human Brain Project Summit 2023},
keywords = {streamflow, across, eupex, space},
month = mar,
note = {Invited talk},
title = {Standardised Workflows at {EBRAINS}},
url = {https://datacloud.di.unito.it/index.php/s/K5YQKTsX9N7NLT8},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/K5YQKTsX9N7NLT8}} CWL for HPC: are we there yet?
Iacopo Colonnelli
across eupex streamflow
Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWL's readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.
@misc{23:CWLConference,
abstract = {Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWL's readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.},
address = {Heidelberg, Germany},
annote = {invited},
author = {Iacopo Colonnelli},
howpublished = {2023 CWL Conference},
keywords = {streamflow, across, eupex},
month = mar,
note = {Invited talk},
title = {{CWL} for {HPC}: are we there yet?},
url = {https://datacloud.di.unito.it/index.php/s/CMCd5LiZeXsxwEg},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/CMCd5LiZeXsxwEg}} Workflows and the Common Workflow Language (CWL)
Iacopo Colonnelli
streamflow
@misc{23:SA2Micro,
address = {Torino, Italy},
author = {Iacopo Colonnelli},
howpublished = {OSA2Micro: An Open Science Approach to Microbiology data integration},
keywords = {streamflow},
month = may,
note = {Invited talk},
title = {Workflows and the Common Workflow Language ({CWL})},
url = {https://datacloud.di.unito.it/index.php/s/NHWWzMMaQgAsA52},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/NHWWzMMaQgAsA52}} UNITO tools presentation
Iacopo Colonnelli
jupyter-workflow streamflow
@misc{23:FL3WorkingDay,
address = {Bologna, Italy},
author = {Iacopo Colonnelli},
howpublished = {CN HPC Flagship 3 Working Day},
keywords = {streamflow, jupyter-workflow},
month = may,
title = {{UNITO} tools presentation},
url = {https://datacloud.di.unito.it/index.php/s/fgHbnLDQSFtcwLd},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/fgHbnLDQSFtcwLd}} Porting the Variant Calling Pipeline for NGS data in cloud-HPC environment
Alberto Mulone, Sherine Awad, Davide Chiarugi, Marco Aldinucci
across icsc streamflow
In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.
@misc{23:mulone:wide:talk,
abstract = {In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.},
address = {Torino, Italy},
author = {Alberto Mulone and Sherine Awad and Davide Chiarugi and Marco Aldinucci},
howpublished = {47th {IEEE} Annual Computers, Software, and Applications Conference, {COMPSAC} 2023},
keywords = {across, icsc, streamflow},
month = jun,
title = {Porting the {V}ariant {C}alling {P}ipeline for {NGS} data in cloud-{HPC} environment},
url = {https://datacloud.di.unito.it/index.php/s/zNLj3LCZNsNxHwy},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/zNLj3LCZNsNxHwy}} Workflow models for heterogeneous distributed systems
Iacopo Colonnelli
jupyter-workflow streamflow
@misc{23:ITADATABestPhDThesis,
address = {Napoli, Italy},
author = {Iacopo Colonnelli},
howpublished = {2nd Italian Conference on Big Data and Data Science (ITADATA 2023)},
keywords = {streamflow, jupyter-workflow},
month = sep,
note = {Best PhD Thesis Award},
title = {Workflow models for heterogeneous distributed systems},
url = {https://datacloud.di.unito.it/index.php/s/6RqcaJ4djqFNDC8},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/6RqcaJ4djqFNDC8}} ACROSS: HPC Big Data Artificial Intelligence Cross Stack Platform Towards Exascale
Iacopo Colonnelli
across streamflow
@misc{23:AssembleaHPC-KTT,
address = {Pisa, Italy},
author = {Iacopo Colonnelli},
howpublished = {LN HPC-KTT Assemblea Nazionale 2023},
keywords = {across, streamflow},
month = oct,
title = {{ACROSS}: {HPC} Big Data Artificial Intelligence Cross Stack Platform Towards Exascale},
url = {https://datacloud.di.unito.it/index.php/s/aK7es8BgFeWorjD},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/aK7es8BgFeWorjD}} Università degli Studi di Torino: Alpha parallel research group
Iacopo Colonnelli, Doriana Medić, Barbara Cantalupo, Marco Aldinucci
icsc streamflow
@misc{23:HaMMonProject,
address = {Bologna, Italy},
author = {Iacopo Colonnelli and Doriana Medi\'{c} and Barbara Cantalupo and Marco Aldinucci},
howpublished = {HaMMon Kick-Off meeting},
keywords = {icsc, streamflow},
month = oct,
title = {Universit\`{a} degli Studi di Torino: Alpha parallel research group},
url = {https://datacloud.di.unito.it/index.php/s/cmgy9BZ3nwCR2QJ},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/cmgy9BZ3nwCR2QJ}} Orchestrating Multi-Domain Workflows: The ACROSS Approach
Alberto Scionti, Iacopo Colonnelli
across streamflow
@misc{23:sc:WCIBoF,
address = {Denver, CO, Usa},
author = {Alberto Scionti and Iacopo Colonnelli},
howpublished = {Workflows Community: Modern Workflows for Continuum and Cross-Facility Computing},
keywords = {across, streamflow},
month = nov,
title = {Orchestrating Multi-Domain Workflows: The {ACROSS} Approach},
url = {https://datacloud.di.unito.it/index.php/s/rJXcDBK4mLmS8yz},
year = 2023,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/rJXcDBK4mLmS8yz}} 2022
2022
StreamFlow: A framework for hybrid workflows
Iacopo Colonnelli
across streamflow
@misc{22:across-streamflow,
address = {Virtual event},
author = {Iacopo Colonnelli},
howpublished = {ACROSS WP4 meeting},
keywords = {streamflow, across},
month = feb,
title = {{StreamFlow}: A framework for hybrid workflows},
url = {https://datacloud.di.unito.it/index.php/s/FXFTKtQSRf6anMX},
year = 2022,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/FXFTKtQSRf6anMX}} StreamFlow
Iacopo Colonnelli
across deephealth eupex streamflow textarossa
@misc{22:healthycloud-workshop,
address = {Virtual event},
annote = {invited},
author = {Iacopo Colonnelli},
howpublished = {2nd HealthyCloud Workshop: Analysis of existing orchestration mechanisms for distributed computational analyses},
keywords = {streamflow, deephealth, across, eupex, textarossa},
month = jul,
note = {Invited talk},
title = {{StreamFlow}},
url = {https://datacloud.di.unito.it/index.php/s/Taz8qtzmkmn9ffT},
year = {2022},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/Taz8qtzmkmn9ffT}} StreamFlow: a topology-aware WMS
Iacopo Colonnelli
across deephealth eupex streamflow textarossa
@misc{22:elixir-streamflow,
address = {Virtual event},
annote = {invited},
author = {Iacopo Colonnelli},
howpublished = {ELIXIR Cloud, Data & AAI Bi-weekly Technical Calls},
keywords = {streamflow, deephealth, across, eupex, textarossa},
month = jun,
note = {Invited talk},
title = {{StreamFlow}: a topology-aware {WMS}},
url = {https://datacloud.di.unito.it/index.php/s/Z9GsKnRCxmBdMd3},
year = {2022},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/Z9GsKnRCxmBdMd3}} OpenDeepHealth: Crafting a Deep Learning Platform as a Service with Kubernetes
Iacopo Colonnelli, Dario Tranchitella
across deephealth hpc4ai jupyter-workflow streamflow
@misc{22:jotb22,
address = {Malaga, Spain},
author = {Iacopo Colonnelli and Dario Tranchitella},
howpublished = {J on The Beach 2022},
keywords = {streamflow, jupyter-workflow, across, deephealth, hpc4ai},
month = apr,
title = {{OpenDeepHealth}: Crafting a Deep Learning Platform as a Service with {K}ubernetes},
url = {https://datacloud.di.unito.it/index.php/s/n6J7STNnwdyqtET},
year = {2022},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/n6J7STNnwdyqtET}} Distributed workflows with Jupyter
Iacopo Colonnelli
across deephealth jupyter-workflow streamflow
@misc{22:jotb22-workshop,
address = {Malaga, Spain},
author = {Iacopo Colonnelli},
howpublished = {J on The Beach 2022},
keywords = {streamflow, jupyter-workflow, deephealth, across},
month = apr,
note = {Workshop},
title = {Distributed workflows with {J}upyter},
url = {https://datacloud.di.unito.it/index.php/s/om89q55S6ePf2Ji},
year = {2022},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/om89q55S6ePf2Ji}} 2021
2021
The Universal Cloud-HPC Pipeline for the AI-Assisted Explainable Diagnosis of COVID-19 Pneumonia
Marco Aldinucci, Iacopo Colonnelli
deephealth hpc4ai streamflow
We'll present a methodology to run DNN pipelines on hybrid cloud+HPC infrastructure. We'll also define a "universal pipeline" for medical images. The pipeline can reproduce all state-of-the-art DNNs to diagnose COVID-19 pneumonia, which appeared in the literature during the first Italian lockdown and following months. We can run all of them (across cloud+HPC platforms) and compare their performance in terms of sensitivity and specificity to set a baseline to evaluate future progress in the automated diagnosis of COVID-19. Also, the pipeline makes existing DNNs explainable by way of adversarial training. The pipeline is easily portable and can run across different infrastructures, adapting the performance-urgency trade-off. The methodology builds onto two novel software programs: the streamflow workflow system and the AI-sandbox concept (parallel container with user-space encrypted file system). We reach over 92% accuracy in diagnosing COVID pneumonia.
@misc{21:gtc:clairecovid,
abstract = {We'll present a methodology to run DNN pipelines on hybrid cloud+HPC infrastructure. We'll also define a "universal pipeline" for medical images. The pipeline can reproduce all state-of-the-art DNNs to diagnose COVID-19 pneumonia, which appeared in the literature during the first Italian lockdown and following months. We can run all of them (across cloud+HPC platforms) and compare their performance in terms of sensitivity and specificity to set a baseline to evaluate future progress in the automated diagnosis of COVID-19. Also, the pipeline makes existing DNNs explainable by way of adversarial training. The pipeline is easily portable and can run across different infrastructures, adapting the performance-urgency trade-off. The methodology builds onto two novel software programs: the streamflow workflow system and the AI-sandbox concept (parallel container with user-space encrypted file system). We reach over 92\% accuracy in diagnosing COVID pneumonia.},
address = {Virtual event},
annote = {invited},
author = {Marco Aldinucci and Iacopo Colonnelli},
howpublished = {NVidia GTC'21},
keywords = {streamflow, deephealth, hpc4ai},
month = apr,
note = {Invited talk},
title = {The Universal Cloud-{HPC} Pipeline for the {AI}-Assisted Explainable Diagnosis of {COVID-19} Pneumonia},
url = {https://datacloud.di.unito.it/index.php/s/AkQLbPpEEtDzbbm},
year = 2021,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/AkQLbPpEEtDzbbm}} From skeletons to workflows in the cloud-edge era
Marco Aldinucci
across admire deephealth streamflow
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments and that makes it possible to execute multiple sites not sharing a common data space. StreamFlow supports both task and data parallelism and enables the reproducible and scalable execution of workflows, such as AI pipelines, in hybrid cloud-HPC environments. As a running example, we use the novel “universal COVID-19 pipeline” that explore the whole optimisation space of the training of different DNNs to classify COVID-19 lung lesions.
@misc{21:hlpp:streamflow,
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments and that makes it possible to execute multiple sites not sharing a common data space. StreamFlow supports both task and data parallelism and enables the reproducible and scalable execution of workflows, such as AI pipelines, in hybrid cloud-HPC environments. As a running example, we use the novel ``universal COVID-19 pipeline'' that explore the whole optimisation space of the training of different DNNs to classify COVID-19 lung lesions.},
address = {Virtual event},
annote = {keynote},
author = {Marco Aldinucci},
howpublished = {14th Intl. Symposium on High-Level Programming and Applications (HLPP)},
keywords = {streamflow, deephealth, across, admire},
month = jul,
note = {Keynote talk},
title = {From skeletons to workflows in the cloud-edge era},
url = {https://datacloud.di.unito.it/index.php/s/RyRPjNBse5PKnab},
year = 2021,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/RyRPjNBse5PKnab}} The modernization of HPC applications for the cloud era
Marco Aldinucci
across admire deephealth streamflow
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g., clouds, supercomputers, and both of them. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments (such as Kubernetes and SLURM), making it possible to execute onto multiple sites not sharing a common data space. Streamflow clearly distinguishes it from many other workflow management systems because it decouples the data dependencies from the deployment of (containerized) workflow steps. Streamflow also leverages CAPIO (Cross-Application Programmable I/O) to move data from one step to another efficiently. CAPIO captures the POSIX file system and streams it in parallel and in-memory to the workflow's next step, possibly enabling in-transit data filtering.
@misc{21:eni:streamflow,
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g., clouds, supercomputers, and both of them. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments (such as Kubernetes and SLURM), making it possible to execute onto multiple sites not sharing a common data space. Streamflow clearly distinguishes it from many other workflow management systems because it decouples the data dependencies from the deployment of (containerized) workflow steps. Streamflow also leverages CAPIO (Cross-Application Programmable I/O) to move data from one step to another efficiently. CAPIO captures the POSIX file system and streams it in parallel and in-memory to the workflow's next step, possibly enabling in-transit data filtering.},
address = {Virtual event},
annote = {keynote},
author = {Marco Aldinucci},
howpublished = {Fifth EAGE Workshop on High Performance Computing for Upstream},
keywords = {streamflow, deephealth, across, admire},
month = sep,
note = {Keynote talk},
title = {The modernization of {HPC} applications for the cloud era},
year = 2021} 2020
2020
StreamFlow: cross breeding cloud with HPC
Iacopo Colonnelli
streamflow
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.
@misc{20:HPCEuropa3TAM,
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.},
address = {Virtual event},
annote = {invited},
author = {Iacopo Colonnelli},
bdsk-utl-2 = {https://drive.google.com/file/d/1aTVhlsrS7R2FzEWRTTtm4Mzr7O8R4Jq1/view},
howpublished = {HPC-Europa3 2nd Transnational Access Meeting (TAM)},
keywords = {streamflow},
month = oct,
note = {Invited talk},
title = {{StreamFlow}: cross breeding cloud with {HPC}},
url = {https://datacloud.di.unito.it/index.php/s/qPHHrSNxk8QXJDw},
year = {2020},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/qPHHrSNxk8QXJDw}} 2019
2019
StreamFlow: un approccio dichiarativo a workflow e pipeline di micro-servizi
Iacopo Colonnelli
streamflow
Negli ultimi anni, gli approcci orientati ai container si sono dimostrati particolarmente efficaci nel garantire portabilità e riproducibilità dei workflow scientifici. Tuttavia, con il continuo aumento del volume di dati a disposizione e la crescente complessità delle procedure di analisi in ogni campo della ricerca, anche i requisiti di performance e riusabilità si fanno via via sempre più essenziali. L'obiettivo principale di StreamFlow è quello di fornire un nuovo paradigma, totalmente dichiarativo, per la descrizione e l'accelerazione di workflow scientifici in ambienti distribuiti. La peculiarità di StreamFlow risiede nel fatto che l'ambiente di esecuzione è interamente descritto in termini di servizi (container), connessioni tra essi e fattori di replica. Inoltre, ogni task del workflow è esplicitamente mappato sulla tipologia di servizio richiesta. Questo permette un maggior controllo sull'utilizzo delle risorse e politiche di scheduling più precise, a vantaggio delle performance. I principali vantaggi di un approccio dichiarativo sono invece la più facile comprensione ed estensione dei modelli esistenti, a vantaggio della riusabilitià.
@misc{19:GarrWorkshop,
abstract = {Negli ultimi anni, gli approcci orientati ai container si sono dimostrati particolarmente efficaci nel garantire portabilit\`{a} e riproducibilit\`{a} dei workflow scientifici. Tuttavia, con il continuo aumento del volume di dati a disposizione e la crescente complessit\`{a} delle procedure di analisi in ogni campo della ricerca, anche i requisiti di performance e riusabilit\`{a} si fanno via via sempre pi\`{u} essenziali. L'obiettivo principale di StreamFlow \`{e} quello di fornire un nuovo paradigma, totalmente dichiarativo, per la descrizione e l'accelerazione di workflow scientifici in ambienti distribuiti. La peculiarit\`{a} di StreamFlow risiede nel fatto che l'ambiente di esecuzione \`{e} interamente descritto in termini di servizi (container), connessioni tra essi e fattori di replica. Inoltre, ogni task del workflow \`{e} esplicitamente mappato sulla tipologia di servizio richiesta. Questo permette un maggior controllo sull'utilizzo delle risorse e politiche di scheduling pi\`{u} precise, a vantaggio delle performance. I principali vantaggi di un approccio dichiarativo sono invece la pi\`{u} facile comprensione ed estensione dei modelli esistenti, a vantaggio della riusabiliti\`{a}.},
address = {Roma, Italy},
author = {Iacopo Colonnelli},
howpublished = {Workshop GARR 2019},
keywords = {streamflow},
month = oct,
title = {{StreamFlow}: un approccio dichiarativo a workflow e pipeline di micro-servizi},
url = {https://datacloud.di.unito.it/index.php/s/kZqyiQnBEQNdXJe},
year = {2019},
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/kZqyiQnBEQNdXJe},
bdsk-url-2 = {https://www.eventi.garr.it/it/ws19/programma/speaker/580-iacopo-colonnelli}}
CAPIO
A joint project with CS Dept. – University of Pisa
CAPIO (Cross-Application Programmable I/O) is a middleware capable of transparently injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without modifying the business code. The contribution is twofold: at design time, a new I/O coordination language allows users to annotate workflow data dependencies with synchronization semantics; at run time, a user-space software layer automatically turns a batch execution into a streaming execution according to the semantics expressed in the configuration file.
CAPIO is a libre software available on Github under the MIT license.
2026
2026
The Italian pathfinding strategy for RISC-V-based HPC: Progress within DARE-SGA1 project
Giovanni Agosta, Marco Aldinucci, Marcello Barbirotta, Andrea Bartolini, Laura Bellentani, Andrea Biagioni, Daniele Cattaneo, Daniele Cesarini, Carlotta Chiarini, Iacopo Colonnelli, Francesco Conti, Pietro Delugas, Lev Denisov, Ottorino Frezza, Angelo Garofalo, Marco Grangetto, Juan David Guerrero Balaguera, Francesca Lo Cicero, Alessandro Lonardo, Michele Martinelli, Andrea Maslov, Francesco Menichelli, Mauro Olivieri, Pierpaolo Perticaroli, Luca Pontisso, Cristian Rossi, Josie Esteban Rodriguez Condia, Daniele Rossi, Davide Rossi, Sergio Saponara, Antonio Sciarappa, Francesco Simula, Matteo Sonza Reorda, Gabriele Spadaro, Massimo Torquati, Piero Vicini
Microprocessors and Microsystems
capio dare jupyter-workflow streamflow
Driven by the need of technological sovereignty, the European Union has accelerated the development of a sustainable High-Performance Computing (HPC) ecosystem. In this perspective, the DARE SGA1 (Digital Autonomy with RISC-V in Europe) project seeks to decouple European infrastructure from proprietary, non-European hardware by developing high-performance processors and computing accelerators based on the open-source RISC-V architecture. This paper introduces and reports about the current status of the multifaceted contributions of Italian partners to the critical pathfinding activities within the DARE SGA1 framework aiming at closing the gap between current prototypes and future HPC systems as well as establishing a bridge towards a fully independent European supercomputing supply chain. In particular, the activities focus on (i) novel RISC-V-based accelerator designs optimized for energy efficiency and high throughput and scaling-out capability; (ii) improvement of RISC-V-based system reliability and resiliency; (iii) evolution of the software stack for low level software and compiler optimizations (iv) co-design through benchmarking with AI and Quantum Chemistry scalable applications.
@article{2026:micpro:dare,
abstract = {Driven by the need of technological sovereignty, the European Union has accelerated the development of a sustainable High-Performance Computing (HPC) ecosystem. In this perspective, the DARE SGA1 (Digital Autonomy with RISC-V in Europe) project seeks to decouple European infrastructure from proprietary, non-European hardware by developing high-performance processors and computing accelerators based on the open-source RISC-V architecture. This paper introduces and reports about the current status of the multifaceted contributions of Italian partners to the critical pathfinding activities within the DARE SGA1 framework aiming at closing the gap between current prototypes and future HPC systems as well as establishing a bridge towards a fully independent European supercomputing supply chain. In particular, the activities focus on (i) novel RISC-V-based accelerator designs optimized for energy efficiency and high throughput and scaling-out capability; (ii) improvement of RISC-V-based system reliability and resiliency; (iii) evolution of the software stack for low level software and compiler optimizations (iv) co-design through benchmarking with AI and Quantum Chemistry scalable applications.},
author = {Giovanni Agosta and Marco Aldinucci and Marcello Barbirotta and Andrea Bartolini and Laura Bellentani and Andrea Biagioni and Daniele Cattaneo and Daniele Cesarini and Carlotta Chiarini and Iacopo Colonnelli and Francesco Conti and Pietro Delugas and Lev Denisov and Ottorino Frezza and Angelo Garofalo and Marco Grangetto and Juan David Guerrero Balaguera and Francesca {Lo Cicero} and Alessandro Lonardo and Michele Martinelli and Andrea Maslov and Francesco Menichelli and Mauro Olivieri and Pierpaolo Perticaroli and Luca Pontisso and Cristian Rossi and Josie Esteban Rodriguez Condia and Daniele Rossi and Davide Rossi and Sergio Saponara and Antonio Sciarappa and Francesco Simula and Matteo Sonza Reorda and Gabriele Spadaro and Massimo Torquati and Piero Vicini},
doi = {10.1016/j.micpro.2026.105321},
issn = {0141-9331},
journal = {Microprocessors and Microsystems},
keywords = {dare, streamflow, jupyter-workflow, capio},
pages = {105321},
title = {The Italian pathfinding strategy for {RISC-V}-based {HPC}: Progress within {DARE-SGA1} project},
url = {https://iris.unito.it/retrieve/e6b5b8a2-a130-4123-9f3b-22f2befeadd0/DARE_ICSC_IEEE_MICPRO.pdf},
volume = {124},
year = 2026,
bdsk-url-1 = {https://iris.unito.it/retrieve/e6b5b8a2-a130-4123-9f3b-22f2befeadd0/DARE_ICSC_IEEE_MICPRO.pdf},
bdsk-url-2 = {https://doi.org/10.1016/j.micpro.2026.105321}} Dynamic transparent streaming in file-based workflows with CAPIO
Marco Edoardo Santimaria, Iacopo Colonnelli, Barbara Cantalupo, Massimo Torquati, Doriana Medić, Nicola Tuccari, Eva Sciacca, Marco Aldinucci
Future Generation Computer Systems
capio eupex icsc space
Advances in big data and the growth in complexity of modern applications highlight the necessity for optimizing workflow executions on different levels, such as hybrid workflow executions, automatic optimization of data movements, and efficient use of IO. Following this line, streaming features are the desired capabilities for file-based workflows as they can reduce overall execution times. Expanding workflows with streaming capabilities usually requires rewriting the application, which is time-consuming and requires deep knowledge of the application. With this work, we introduce the Cross-Application Programmable IO (CAPIO) methodology, of which the stack is composed of two parts: the CAPIO-CL coordination language and the CAPIO middleware (which implements the semantics expressed by the CAPIO-CL coordination language). The CAPIO-CL coordination language annotates synchronization semantics between files produced and consumed by workflow steps. At the same time, the CAPIO middleware improves the performance of file-based workflows, leveraging the information provided by the CAPIO-CL language while not having to change (recompile) the code of the original workflow steps. By design, the CAPIO middleware supports multiple backends and can be extended to support more. It is dynamic, and it supports dynamic job scheduling. Benchmarks, done on both microbenchmarks and real-life workflows, prove that with CAPIO, it is possible to reduce the workflow execution time by up to 50
@article{25:fgcs:capio,
abstract = {Advances in big data and the growth in complexity of modern applications highlight the necessity for optimizing workflow executions on different levels, such as hybrid workflow executions, automatic optimization of data movements, and efficient use of IO. Following this line, streaming features are the desired capabilities for file-based workflows as they can reduce overall execution times. Expanding workflows with streaming capabilities usually requires rewriting the application, which is time-consuming and requires deep knowledge of the application. With this work, we introduce the Cross-Application Programmable IO (CAPIO) methodology, of which the stack is composed of two parts: the CAPIO-CL coordination language and the CAPIO middleware (which implements the semantics expressed by the CAPIO-CL coordination language). The CAPIO-CL coordination language annotates synchronization semantics between files produced and consumed by workflow steps. At the same time, the CAPIO middleware improves the performance of file-based workflows, leveraging the information provided by the CAPIO-CL language while not having to change (recompile) the code of the original workflow steps. By design, the CAPIO middleware supports multiple backends and can be extended to support more. It is dynamic, and it supports dynamic job scheduling. Benchmarks, done on both microbenchmarks and real-life workflows, prove that with CAPIO, it is possible to reduce the workflow execution time by up to 50%},
author = {Santimaria, Marco Edoardo and Colonnelli, Iacopo and Cantalupo, Barbara and Torquati, Massimo and Medi\'{c}, Doriana and Tuccari, Nicola and Sciacca, Eva and Aldinucci, Marco},
date-modified = {2026-08-14 12:42:32 +0200},
doi = {10.1016/j.future.2025.108159},
issn = {0167-739X},
journal = {Future Generation Computer Systems},
keywords = {capio, eupex, space, icsc},
month = {March},
pages = {108159},
publisher = {Elsevier BV},
title = {Dynamic transparent streaming in file-based workflows with CAPIO},
url = {http://dx.doi.org/10.1016/j.future.2025.108159},
year = {2026},
bdsk-url-1 = {http://dx.doi.org/10.1016/j.future.2025.108159}} 2025
2025
Towards RISC-V-based HPC: The Italian Pathfinding Activities in the DARE-SGA1 Project
Giovanni Agosta, Marco Aldinucci, Andrea Bartolini, Laura Bellentani, Andrea Biagioni, Daniele Cesarini, Carlotta Chiarini, Iacopo Colonnelli, Pietro Delugas, Lev Denisov, Ottorino Frezza, Marco Grangetto, Francesca Lo Cicero, Alessandro Lonardo, Michele Martinelli, Andrea Maslov, Mauro Olivieri, Pierpaolo Perticaroli, Luca Pontisso, Cristian Rossi, Davide Rossi, Sergio Saponara, Antonio Sciarappa, Francesco Simula, Matteo Sonza Reorda, Massimo Torquati, Piero Vicini
28th Euromicro Conference on Digital System Design, DSD 2025
capio dare streamflow
The European Union's efforts towards technological sovereignty in High-Performance Computing are driving research and development of RISC-V-based supercomputers. The DARE SGA1 project, in particular, aims to develop chips designed and owned by Europeans. This paper introduces the Italian contribution to DARE SGA1 regarding pathfinding activities toward future RISC-V-based accelerator designs, reliability improvements, system software, and AI and Quantum Chemistry applications.
@inproceedings{2025:dsd:dare,
abstract = {The European Union's efforts towards technological sovereignty in High-Performance Computing are driving research and development of RISC-V-based supercomputers. The DARE SGA1 project, in particular, aims to develop chips designed and owned by Europeans. This paper introduces the Italian contribution to DARE SGA1 regarding pathfinding activities toward future RISC-V-based accelerator designs, reliability improvements, system software, and AI and Quantum Chemistry applications.},
author = {Agosta, Giovanni and Aldinucci, Marco and Bartolini, Andrea and Bellentani, Laura and Biagioni, Andrea and Cesarini, Daniele and Chiarini, Carlotta and Colonnelli, Iacopo and Delugas, Pietro and Denisov, Lev and Frezza, Ottorino and Grangetto, Marco and Lo Cicero, Francesca and Lonardo, Alessandro and Martinelli, Michele and Maslov, Andrea and Olivieri, Mauro and Perticaroli, Pierpaolo and Pontisso, Luca and Rossi, Cristian and Rossi, Davide and Saponara, Sergio and Sciarappa, Antonio and Simula, Francesco and Sonza Reorda, Matteo and Torquati, Massimo and Vicini, Piero},
booktitle = {28th Euromicro Conference on Digital System Design, {DSD} 2025},
doi = {10.1109/DSD67783.2025.00057},
keywords = {dare, streamflow, capio},
location = {Salerno, Italy},
mon = {sep},
pages = {360--367},
publisher = {{IEEE}},
title = {Towards {RISC-V}-based {HPC:} The Italian Pathfinding Activities in the {DARE-SGA1} Project},
url = {https://iris.unito.it/retrieve/03658af6-782d-4f9c-9108-b0fc1f1859eb/DARE_ICSC_EPDSD2025-1.pdf},
year = {2025},
bdsk-url-1 = {https://iris.unito.it/retrieve/03658af6-782d-4f9c-9108-b0fc1f1859eb/DARE_ICSC_EPDSD2025-1.pdf},
bdsk-url-2 = {https://doi.org/10.1109/DSD67783.2025.00057}} Overcoming Dynamic I/O Boundaries: a Double-Sided Streaming Methodology with dispel4py and CAPIO
Marco Edoardo Santimaria, Rosa Filgueira, Doriana Medić, Iacopo Colonnelli, Marco Aldinucci
Proceedings of the SC 25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis
capio dare icsc
This work introduces a novel double-sided streaming methodology that combines control-plane and data-plane streaming. Our goal is to implement the long-advocated separation of concerns in workflow orchestration without introducing artificial boundaries in their execution. Our approach is exemplified by the integration of control-plane streaming provided by dispel4py and the transparent data-plane streaming provided by CAPIO. Our integration eliminates file synchronization barriers without requiring modifications to existing workflow logic. To support this, we extend CAPIO with a new commit rule that allows streaming over dynamically generated file sets, enabling hybrid workflows that blend in-memory dataflows with file-based communication. We validate our approach using a real-world seismic cross-correlation workflow, achieving performance improvements between 23
@inproceedings{Santimaria:works:25,
abstract = {This work introduces a novel double-sided streaming methodology that combines control-plane and data-plane streaming. Our goal is to implement the long-advocated separation of concerns in workflow orchestration without introducing artificial boundaries in their execution. Our approach is exemplified by the integration of control-plane streaming provided by dispel4py and the transparent data-plane streaming provided by CAPIO. Our integration eliminates file synchronization barriers without requiring modifications to existing workflow logic. To support this, we extend CAPIO with a new commit rule that allows streaming over dynamically generated file sets, enabling hybrid workflows that blend in-memory dataflows with file-based communication. We validate our approach using a real-world seismic cross-correlation workflow, achieving performance improvements between 23% and 40%. Unlike previous solutions, our method supports streaming across the entire workflow, including phase boundaries where file I/O would typically enforce strict execution ordering. Therefore, our approach can be straightforwardly extended to other multi-stage streaming applications.},
author = {Santimaria, Marco Edoardo and Filgueira, Rosa and Medi\'{c}, Doriana and Colonnelli, Iacopo and Aldinucci, Marco},
booktitle = {Proceedings of the SC 25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis},
collection = {SC Workshops - 25},
doi = {10.1145/3731599.3767577},
keywords = {capio,dare,icsc},
month = nov,
pages = {2269--2280},
publisher = {ACM},
series = {SC Workshops 25},
title = {Overcoming Dynamic I/O Boundaries: a Double-Sided Streaming Methodology with dispel4py and CAPIO},
url = {http://dx.doi.org/10.1145/3731599.3767577},
year = {2025},
bdsk-url-1 = {http://dx.doi.org/10.1145/3731599.3767577}} Directed Acyclic Graph on Cross-Application Programmable I/O: Adding streaming flavour to scientific workflows
Simone Perrotta, Marco Edoardo Santimaria, Ciro Giuseppe De Vita, Massimo Torquati, Diana Di Luccio, Pasquale Corvino, Antonella Pirozzi, Raffaele Montella
2025 IEEE International Conference on eScience (eScience)
capio icsc
This paper introduces DAGonCAPIO, a workflow framework that integrates the DAGonStar engine with the CAPIO middleware to enable I/O streaming in scientific workflows. DAGonStar utilizes a Directed Acyclic Graph (DAG) to orchestrate tasks. At the same time, CAPIO enables downstream tasks to process data as soon as partial outputs become available, without requiring modifications to application code. This integration reduces delays from traditional file-based communication. The system uses the workflow:// schema to define data dependencies and generate CAPIO coordination scripts. DAGonStar was modified to support early scratch directory naming and decoupled task execution. Experiments with a WRF-based weather forecasting workflow on a 256-core cluster demonstrate that DAGonCAPIO reduces time-to-first-result by up to 4171 seconds, achieving a nearly 10x speedup.
@inproceedings{25:dagoncapio:escience,
abstract = {This paper introduces DAGonCAPIO, a workflow framework that integrates the DAGonStar engine with the CAPIO middleware to enable I/O streaming in scientific workflows. DAGonStar utilizes a Directed Acyclic Graph (DAG) to orchestrate tasks. At the same time, CAPIO enables downstream tasks to process data as soon as partial outputs become available, without requiring modifications to application code. This integration reduces delays from traditional file-based communication. The system uses the workflow:// schema to define data dependencies and generate CAPIO coordination scripts. DAGonStar was modified to support early scratch directory naming and decoupled task execution. Experiments with a WRF-based weather forecasting workflow on a 256-core cluster demonstrate that DAGonCAPIO reduces time-to-first-result by up to 4171 seconds, achieving a nearly 10x speedup.},
author = {Perrotta, Simone and Santimaria, Marco Edoardo and De Vita, Ciro Giuseppe and Torquati, Massimo and Di Luccio, Diana and Corvino, Pasquale and Pirozzi, Antonella and Montella, Raffaele},
booktitle = {2025 IEEE International Conference on eScience (eScience)},
doi = {10.1109/escience65000.2025.00047},
keywords = {capio,icsc},
month = sep,
pages = {313--314},
publisher = {IEEE},
title = {Directed Acyclic Graph on Cross-Application Programmable I/O: Adding streaming flavour to scientific workflows},
url = {http://dx.doi.org/10.1109/eScience65000.2025.00047},
year = {2025},
bdsk-url-1 = {http://dx.doi.org/10.1109/eScience65000.2025.00047}} Streaming I/O for scientific workflow engine acceleration
Simone Perrotta, Ciro Giuseppe De Vita, Gennaro Mellone, Marco Edoardo Santimaria, Massimo Torquati, Javier Garcia Blas, Raffaele Montella
Future Generation Computer Systems
capio
Scientific workflows are increasingly characterized by complex task dependencies and large-scale data exchanges, which place significant pressure on the input/output (I/O) systems of traditional Workflow Engines (WFEs). These challenges are particularly evident in data-intensive and real-time processing contexts, where conventional disk-based I/O mechanisms often become performance bottlenecks. This paper presents an approach to enhancing the DAGonStar scientific workflow engine by integrating CAPIO, a middleware designed to support memory-based streaming I/O. The integration combines DAGonStar's orchestration capabilities with CAPIO's efficient data handling to better support workflows operating on continuous or large-scale datasets. We describe the architectural modifications introduced to enable this collaboration and provide an analysis of the resulting system. The proposed solution aims to improve the responsiveness and flexibility of scientific workflows by streamlining data transfers and simplifying task coordination. This work contributes to the evolution of workflow systems toward more efficient and scalable models for scientific computing.
@article{25:FGCS:DagonCapio,
abstract = {Scientific workflows are increasingly characterized by complex task dependencies and large-scale data exchanges, which place significant pressure on the input/output (I/O) systems of traditional Workflow Engines (WFEs). These challenges are particularly evident in data-intensive and real-time processing contexts, where conventional disk-based I/O mechanisms often become performance bottlenecks. This paper presents an approach to enhancing the DAGonStar scientific workflow engine by integrating CAPIO, a middleware designed to support memory-based streaming I/O. The integration combines DAGonStar's orchestration capabilities with CAPIO's efficient data handling to better support workflows operating on continuous or large-scale datasets. We describe the architectural modifications introduced to enable this collaboration and provide an analysis of the resulting system. The proposed solution aims to improve the responsiveness and flexibility of scientific workflows by streamlining data transfers and simplifying task coordination. This work contributes to the evolution of workflow systems toward more efficient and scalable models for scientific computing.},
author = {Perrotta, Simone and De Vita, Ciro Giuseppe and Mellone, Gennaro and Santimaria, Marco Edoardo and Torquati, Massimo and Garcia Blas, Javier and Montella, Raffaele},
doi = {10.1016/j.future.2025.107978},
issn = {0167-739X},
journal = {Future Generation Computer Systems},
keywords = {capio,},
pages = {107978},
title = {Streaming I/O for scientific workflow engine acceleration},
url = {https://www.sciencedirect.com/science/article/pii/S0167739X25002730},
year = {2025},
bdsk-url-1 = {https://www.sciencedirect.com/science/article/pii/S0167739X25002730},
bdsk-url-2 = {https://doi.org/10.1016/j.future.2025.107978}} Extending a Scientific Workflow Engine with Streaming I/O Capabilities: DAGonStar and CAPIO
Simone Perrotta, Ciro Giuseppe De Vita, Gennaro Mellone, Marco Edoardo Santimaria, Giuseppe Salvi, Marco Lapegna, Massimo Torquati, Angelo Ciaramella
Euro-Par 2024: Parallel Processing Workshops
capio
The increasing complexity and scale of data-intensive scientific workflows necessitate advancements in workflow engines (WFEs) to handle real-time data streams and reduce input/output (I/O) bottlenecks. This paper introduces an innovative approach to enhancing the DAGonStar scientific workflow engine by integrating CAPIO, a middleware capable of injecting I/O streaming capabilities into traditional scientific workflows and optimized for high-speed data access and low latency. By combining DAGonStar's robust task orchestration and dependency management with CAPIO, we aim to significantly improve scientific workflows' performance and scalability. We present the design and implementation of this integration, detailing the architectural modifications required to enable seamless interaction between DAGonStar and CAPIO. The paper includes comprehensive benchmarks and performance evaluations demonstrating the impact of CAPIO on workflow execution times and data handling efficiency. Our findings indicate that the enhanced DAGonStar, equipped with CAPIO, offers a powerful solution for managing and processing large-scale, real-time data streams, thereby advancing the capabilities of scientific computing infrastructure.
@inproceedings{25:dagon-capio-santimaria,
abstract = {The increasing complexity and scale of data-intensive scientific workflows necessitate advancements in workflow engines (WFEs) to handle real-time data streams and reduce input/output (I/O) bottlenecks. This paper introduces an innovative approach to enhancing the DAGonStar scientific workflow engine by integrating CAPIO, a middleware capable of injecting I/O streaming capabilities into traditional scientific workflows and optimized for high-speed data access and low latency. By combining DAGonStar's robust task orchestration and dependency management with CAPIO, we aim to significantly improve scientific workflows' performance and scalability. We present the design and implementation of this integration, detailing the architectural modifications required to enable seamless interaction between DAGonStar and CAPIO. The paper includes comprehensive benchmarks and performance evaluations demonstrating the impact of CAPIO on workflow execution times and data handling efficiency. Our findings indicate that the enhanced DAGonStar, equipped with CAPIO, offers a powerful solution for managing and processing large-scale, real-time data streams, thereby advancing the capabilities of scientific computing infrastructure.},
address = {Cham},
author = {Perrotta, Simone and De Vita, Ciro Giuseppe and Mellone, Gennaro and Santimaria, Marco Edoardo and Salvi, Giuseppe and Lapegna, Marco and Torquati, Massimo and Ciaramella, Angelo},
booktitle = {Euro-Par 2024: Parallel Processing Workshops},
doi = {10.1007/978-3-031-90203-1_12},
isbn = {978-3-031-90203-1},
keywords = {capio},
pages = {129--140},
publisher = {Springer Nature Switzerland},
title = {Extending a Scientific Workflow Engine with Streaming I/O Capabilities: DAGonStar and CAPIO},
url = {https://link.springer.com/chapter/10.1007/978-3-031-90203-1_12},
year = {2025},
bdsk-url-1 = {https://link.springer.com/chapter/10.1007/978-3-031-90203-1_12},
bdsk-url-2 = {https://doi.org/10.1007/978-3-031-90203-1_12}} CAPIO-CL: The CAPIO Coordination Language
Marco Edoardo Santimaria, Alberto R. Martinelli, Iacopo Colonnelli, Barbara Cantalupo, Massimo Torquati, Marco Aldinucci
International Journal of Parallel Programming
capio icsc
The performance bottleneck in file-based workflows remains a pressing issue in the realm of I/O-based workflows. To address this challenge, a novel annotation language has been developed. CAPIO-CL is positioned as an innovative I/O coordination language, enabling users to annotate data dependencies within file-based workflows with synchronization semantics pertinent to the involved files and directories. Through the information provided by the language, optimization opportunities arise in streaming and preemptive data movement. This paper serves to illustrate the semantics and syntax enabling CAPIO-CL to enhance the performance of in situ workflows without necessitating the rewriting or modification of the original workflow application steps. Finally, an analysis of CAPIO-CL is provided, taking into consideration both language expressiveness and application performance enhancement.
@inproceedings{santimaria_capio_cl_25,
abstract = {The performance bottleneck in file-based workflows remains a pressing issue in the realm of I/O-based workflows. To address this challenge, a novel annotation language has been developed. CAPIO-CL is positioned as an innovative I/O coordination language, enabling users to annotate data dependencies within file-based workflows with synchronization semantics pertinent to the involved files and directories. Through the information provided by the language, optimization opportunities arise in streaming and preemptive data movement. This paper serves to illustrate the semantics and syntax enabling CAPIO-CL to enhance the performance of in situ workflows without necessitating the rewriting or modification of the original workflow application steps. Finally, an analysis of CAPIO-CL is provided, taking into consideration both language expressiveness and application performance enhancement.},
author = {Santimaria, Marco Edoardo and Martinelli, Alberto R. and Colonnelli, Iacopo and Cantalupo, Barbara and Torquati, Massimo and Aldinucci, Marco},
date = {2025/02/25},
date-added = {2025-02-25 19:59:05 +0100},
date-modified = {2025-02-25 20:00:33 +0100},
doi = {10.1007/s10766-025-00789-0},
id = {Santimaria2025},
isbn = {1573-7640},
journal = {International Journal of Parallel Programming},
keywords = {capio, icsc},
number = {2},
pages = {10},
title = {CAPIO-CL: The CAPIO Coordination Language},
url = {https://doi.org/10.1007/s10766-025-00789-0},
volume = {53},
year = {2025},
bdsk-url-1 = {https://doi.org/10.1007/s10766-025-00789-0}} 2023
2023
CAPIO: a Middleware for Transparent I/O Streaming in Data-Intensive Workflows
Alberto R. Martinelli, Massimo Torquati, Marco Aldinucci, Iacopo Colonnelli, Barbara Cantalupo
2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)
admire capio eupex icsc
With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to quickly expand, further exacerbating the performance gap between computing, memory, and storage technologies. This paper introduces CAPIO (Cross-Application Programmable I/O), a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. The contribution is twofold: 1) at design time, a new I/O coordination language allows users to annotate workflow data dependencies with synchronization semantics; 2) at run time, a user-space middleware automatically and transparently to the user turns a workflow batch execution into a streaming execution according to the semantics expressed in the configuration file. CAPIO has been tested on synthetic benchmarks simulating typical workflow I/O patterns and two real-world workflows. Experiments show that CAPIO reduces the execution time by 10% to 66% for data-intensive workflows that use the file system as a communication medium.
@inproceedings{23:hipc:capio,
abstract = {With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to quickly expand, further exacerbating the performance gap between computing, memory, and storage technologies. This paper introduces CAPIO (Cross-Application Programmable I/O), a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. The contribution is twofold: 1) at design time, a new I/O coordination language allows users to annotate workflow data dependencies with synchronization semantics; 2) at run time, a user-space middleware automatically and transparently to the user turns a workflow batch execution into a streaming execution according to the semantics expressed in the configuration file. CAPIO has been tested on synthetic benchmarks simulating typical workflow I/O patterns and two real-world workflows. Experiments show that CAPIO reduces the execution time by 10\% to 66\% for data-intensive workflows that use the file system as a communication medium.},
address = {Goa, India},
author = {Martinelli, Alberto R. and Torquati, Massimo and Aldinucci, Marco and Colonnelli, Iacopo and Cantalupo, Barbara},
booktitle = {2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)},
doi = {10.1109/HiPC58850.2023.00031},
keywords = {admire, eupex, icsc, capio},
month = dec,
publisher = {{IEEE}},
title = {{CAPIO}: a Middleware for Transparent I/O Streaming in Data-Intensive Workflows},
url = {https://iris.unito.it/retrieve/27380f37-0978-409e-a9d8-2b5e95a4bb85/CAPIO-HiPC23-preprint.pdf},
year = {2023},
bdsk-url-1 = {https://iris.unito.it/retrieve/27380f37-0978-409e-a9d8-2b5e95a4bb85/CAPIO-HiPC23-preprint.pdf},
bdsk-url-2 = {https://doi.org/10.1109/HiPC58850.2023.00031}} 2025
2025
Hybrid Workflows in the Heterogeneous Era
Iacopo Colonnelli
capio eupex streamflow
Thanks to their generality, workflow models represent a powerful abstraction for designing complex applications and executing them on large-scale distributed architectures. However, several additional challenges appear when transitioning from cloud/HPC environments to the entire computing continuum. Continuum execution environments are fully distributed and modular, and modules can be heterogeneous and independent of each other. In addition, continuum workflows often rely on multiple intercommunicating agents that form complex micro-services architectures. Different agents deal with different communication and parallelization paradigms: network-based stream processing at the edge and file-based batch processing on HPC facilities. Finally, support for efficient interactive workflows in the continuum remains an open research problem. This webinar delves into these challenges, exploring how current research addresses them with innovatives techniques and tools (i.e., hybrid workflows and StreamFlow, literate workflows and Jupyter Workflow, transparent streaming workflows and CAPIO), and suggesting potential directions for future investigation.
@misc{25:eupex:streamflow,
abstract = {Thanks to their generality, workflow models represent a powerful abstraction for designing complex applications and executing them on large-scale distributed architectures. However, several additional challenges appear when transitioning from cloud/HPC environments to the entire computing continuum. Continuum execution environments are fully distributed and modular, and modules can be heterogeneous and independent of each other. In addition, continuum workflows often rely on multiple intercommunicating agents that form complex micro-services architectures. Different agents deal with different communication and parallelization paradigms: network-based stream processing at the edge and file-based batch processing on HPC facilities. Finally, support for efficient interactive workflows in the continuum remains an open research problem. This webinar delves into these challenges, exploring how current research addresses them with innovatives techniques and tools (i.e., hybrid workflows and StreamFlow, literate workflows and Jupyter Workflow, transparent streaming workflows and CAPIO), and suggesting potential directions for future investigation.},
author = {Iacopo Colonnelli},
keywords = {eupex, streamflow, capio},
month = feb,
title = {Hybrid Workflows in the Heterogeneous Era},
url = {https://datacloud.di.unito.it/index.php/s/Y76mXwCRKJ2CDa3},
year = 2025,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/Y76mXwCRKJ2CDa3}} 2024
2024
CAPIO-CL: Cross Application Programmable IO - Coordination Language
Marco Edoardo Santimaria
across admire capio eupex icsc
The performance bottleneck in file-based workflows remains a pressing issue in the realm of I/O-based workflows. To address this challenge, a novel annotation language has been developed. CAPIO-CL is positioned as an innovative I/O coordination language, enabling users to annotate data dependencies within file-based workflows with synchronization semantics pertinent to the involved files and directories. Through the information provided by the language, optimization opportunities arise in streaming and preemptive data movement. This paper serves to illustrate the semantics and syntax enabling CAPIO-CL to enhance the performance of in situ workflows without necessitating the rewriting or modification of the original workflow application steps. Finally, an analysis of CAPIO-CL is provided, taking into consideration both language expressiveness and application performance enhancement.
@misc{24:santimaria:hlpp:capiocl,
abstract = {The performance bottleneck in file-based workflows remains a pressing issue in the realm of I/O-based workflows. To address this challenge, a novel annotation language has been developed. CAPIO-CL is positioned as an innovative I/O coordination language, enabling users to annotate data dependencies within file-based workflows with synchronization semantics pertinent to the involved files and directories. Through the information provided by the language, optimization opportunities arise in streaming and preemptive data movement. This paper serves to illustrate the semantics and syntax enabling CAPIO-CL to enhance the performance of in situ workflows without necessitating the rewriting or modification of the original workflow application steps. Finally, an analysis of CAPIO-CL is provided, taking into consideration both language expressiveness and application performance enhancement.},
address = {Pisa, Italy},
author = {Marco Edoardo Santimaria},
date-modified = {2026-09-19 19:50:32 +0200},
keywords = {icsc,capio,across,eupex,admire},
month = jul,
title = {{CAPIO-CL}: Cross Application Programmable IO - Coordination Language},
url = {https://datacloud.di.unito.it/index.php/s/zsKY3PWzX5NFCiX},
year = 2024,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/zsKY3PWzX5NFCiX}} Releasing the CAPIO middleware from MPI derived constraints
Marco Edoardo Santimaria, Iacopo Colonnelli, Marco Aldinucci
across admire capio capiocl eupex icsc
CAPIO is a middleware that transparently injects streaming capabilities into file-based workflows. However, its implementation is limited to HPC environments based on the MPI framework, significantly limiting its applications. This paper will illustrate a proposed architecture and some preliminary results aimed at investigating the usage of a distributed files system as a communication media for the CAPIO middleware, with the ultimate goal of supporting both CLOUD-based and HPC-based workflows.
@misc{24:santimaria:bighpc,
abstract = {CAPIO is a middleware that transparently injects streaming capabilities into file-based workflows. However, its implementation is limited to HPC environments based on the MPI framework, significantly limiting its applications. This paper will illustrate a proposed architecture and some preliminary results aimed at investigating the usage of a distributed files system as a communication media for the CAPIO middleware, with the ultimate goal of supporting both CLOUD-based and HPC-based workflows.},
address = {Pisa, Italy},
author = {Santimaria, Marco Edoardo and Colonnelli, Iacopo and Aldinucci, Marco},
keywords = {icsc, capio, capiocl, across, admire, eupex},
month = sep,
title = {Releasing the CAPIO middleware from MPI derived constraints},
url = {https://datacloud.di.unito.it/index.php/s/zrJGD4i36fWdp5g},
year = 2024,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/zrJGD4i36fWdp5g}} CAPIO: Cross Application Programamble IO
Marco Edoardo Santimaria, Iacopo Colonnelli, Massimo Torquati, Marco Aldinucci
across admire capio capiocl eupex icsc
With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to expand, further exacerbating quickly the performance gap between computing, memory, and storage technologies. CAPIO (Cross-Application Programmable I/O), is a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. In this presentation, we will introduce the CAPIO-CL language with its semantics, as well as the implementation of the CAPIO-CL language through the CAPIO middleware. We will also provide some case studies of how CAPIO has been employed to improve workflow execution time as well as some future directions.
@misc{24:santimaria:itadata:shpcpee,
abstract = {With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to expand, further exacerbating quickly the performance gap between computing, memory, and storage technologies. CAPIO (Cross-Application Programmable I/O), is a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. In this presentation, we will introduce the CAPIO-CL language with its semantics, as well as the implementation of the CAPIO-CL language through the CAPIO middleware. We will also provide some case studies of how CAPIO has been employed to improve workflow execution time as well as some future directions.},
address = {Pisa, Italy},
author = {Santimaria, Marco Edoardo and Colonnelli, Iacopo and Torquati, Massimo and Aldinucci, Marco},
keywords = {icsc, capio, capiocl, across, admire, eupex},
month = sep,
title = {CAPIO: Cross Application Programamble IO},
url = {https://datacloud.di.unito.it/index.php/s/rg6LWwrZXi6tTXm},
year = 2024,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/rg6LWwrZXi6tTXm}}
FastFlow
A joint project with CS Dept. – University of Pisa
FastFlow (斋戒流) is a C++ parallel programming framework advocating high-level, pattern-based parallel programming. It chiefly supports streaming and data parallelism, targeting heterogenous platforms composed of clusters of shared-memory platforms, possibly equipped with computing accelerators such as NVidia GPGPUs, Xeon Phi, Tilera TILE64.
At today, FastFlow has been the background technology of 3 European Projects and 1 National project for an aggregate total cost of 12M € (ParaPhrase FP7, REPARA FP7, Rephrase H2020, and IMPACT, see projects section). We are still actively developing FastFlow along with its underlying technology, and we are wide open to turn challenges in research and innovation.
FastFlow comes as a C++ template library designed as a stack of layers that progressively abstracts the programming of parallel applications. The goal of the stack is threefold: portability, extensibility, and performance. For this, all three layers are realized as thin strata of C++ templates that are 1) seamlessly portable, 2) easily extended via subclassing, and 3) statically compiled and cross-optimized with the application. The terse design ensures easy portability on almost all OSes and CPUs with a C++ compiler.
More details in the FastFlow website.
2020
2020
Challenging the abstraction penalty in parallel patterns libraries: Adding FastFlow support to GrPPI
Jose Daniel Garcia, Jose Daniel Rio, Marco Aldinucci, Fabio Tordini, Marco Danelutto, Gabriele Mencagli, Massimo Torquati
The Journal of Supercomputing
fastflow rephrase
In the last years, pattern-based programming has been recognized as a good practice for efficiently exploiting parallel hardware resources. Following this approach, multiple libraries have been designed for providing such high-level abstractions to ease the parallel programming. However, those libraries do not share a common interface. To pave the way, GrPPI has been designed for providing an intermediate abstraction layer between application developers and existing parallel programming frameworks like OpenMP, Intel TBB or ISO C++ threads. On the other hand, FastFlow has been adopted as an efficient object-based programming framework that may benefit from being supported as an additional GrPPI backend. However, the object-based approach presents some major challenges to be incorporated under the GrPPI type safe functional programming style. In this paper, we present the integration of FastFlow as a new GrPPI backend to demonstrate that structured parallel programming frameworks perfectly fit the GrPPI design. Additionally, we also demonstrate that GrPPI does not incur in additional overheads for providing its abstraction layer, and we study the programmability in terms of lines of code and cyclomatic complexity. In general, the presented work acts as reciprocal validation of both FastFlow (as an efficient, native structured parallel programming framework) and GrPPI (as an efficient abstraction layer on top of existing parallel programming frameworks).
@article{19:jsupe:grppi,
abstract = {In the last years, pattern-based programming has been recognized as a good practice for efficiently exploiting parallel hardware resources. Following this approach, multiple libraries have been designed for providing such high-level abstractions to ease the parallel programming. However, those libraries do not share a common interface. To pave the way, GrPPI has been designed for providing an intermediate abstraction layer between application developers and existing parallel programming frameworks like OpenMP, Intel TBB or ISO C++ threads. On the other hand, FastFlow has been adopted as an efficient object-based programming framework that may benefit from being supported as an additional GrPPI backend. However, the object-based approach presents some major challenges to be incorporated under the GrPPI type safe functional programming style. In this paper, we present the integration of FastFlow as a new GrPPI backend to demonstrate that structured parallel programming frameworks perfectly fit the GrPPI design. Additionally, we also demonstrate that GrPPI does not incur in additional overheads for providing its abstraction layer, and we study the programmability in terms of lines of code and cyclomatic complexity. In general, the presented work acts as reciprocal validation of both FastFlow (as an efficient, native structured parallel programming framework) and GrPPI (as an efficient abstraction layer on top of existing parallel programming frameworks).},
author = {Garcia, Jose Daniel and del Rio, Jose Daniel and Aldinucci, Marco and Tordini, Fabio and Danelutto, Marco and Mencagli, Gabriele and Torquati, Massimo},
date-added = {2019-03-23 10:08:27 +0100},
date-modified = {2020-11-15 19:15:18 +0100},
doi = {10.1007/s11227-019-02826-5},
journal = {The Journal of Supercomputing},
keywords = {rephrase, fastflow},
number = {7},
pages = {5139--5159},
title = {Challenging the abstraction penalty in parallel patterns libraries: Adding FastFlow support to GrPPI},
url = {https://iris.unito.it/retrieve/handle/2318/1762686/744894/2020-js-grppi-postprint.pdf},
volume = {76},
year = {2020},
bdsk-url-1 = {https://doi.org/10.1007/s11227-019-02826-5}} 2019
2019
The RePhrase Extended Pattern Set for Data Intensive Parallel Computing
Marco Danelutto, Tiziano De Matteis, Daniele De Sensi, Gabriele Mencagli, Massimo Torquati, Marco Aldinucci, Peter Kilpatrick
International Journal of Parallel Programming
fastflow rephrase
We discuss the extended parallel pattern set identified within the EU-funded project RePhrase as a candidate pattern set to support data intensive applications targeting heterogeneous architectures. The set has been designed to include three classes of pattern, namely i) core patterns, modelling common, not necessarily data intensive parallelism exploitation patterns, usually to be used in composition; ii) high level patterns, modelling common, complex and complete parallelism exploitation patterns; and iii) building block patterns, modelling the single components of data intensive applications, suitable for use–in composition–to implement patterns not covered by the core and high level patterns. We discuss the expressive power of the RePhrase extended pattern set and results illustrating the performances that may be achieved with the FastFlow implementation of the high level patterns.
@article{17:rephrasepatterns:ijpp,
abstract = {We discuss the extended parallel pattern set identified within the EU-funded project RePhrase as a candidate pattern set to support data intensive applications targeting heterogeneous architectures. The set has been designed to include three classes of pattern, namely i) core patterns, modelling common, not necessarily data intensive parallelism exploitation patterns, usually to be used in composition; ii) high level patterns, modelling common, complex and complete parallelism exploitation patterns; and iii) building block patterns, modelling the single components of data intensive applications, suitable for use--in composition--to implement patterns not covered by the core and high level patterns. We discuss the expressive power of the RePhrase extended pattern set and results illustrating the performances that may be achieved with the FastFlow implementation of the high level patterns.},
author = {Danelutto, Marco and De Matteis, Tiziano and De Sensi, Daniele and Mencagli, Gabriele and Torquati, Massimo and Aldinucci, Marco and Kilpatrick, Peter},
date-added = {2017-11-25 15:30:59 +0000},
date-modified = {2019-03-22 23:49:02 +0100},
doi = {10.1007/s10766-017-0540-z},
journal = {International Journal of Parallel Programming},
keywords = {rephrase, fastflow},
number = {1},
pages = {74--93},
title = {The RePhrase Extended Pattern Set for Data Intensive Parallel Computing},
url = {https://iris.unito.it/retrieve/handle/2318/1659336/387667/2017_ijpp_rephrase.pdf},
volume = {47},
year = {2019},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1659336/387667/2017_ijpp_rephrase.pdf},
bdsk-url-2 = {https://doi.org/10.1007/s10766-017-0540-z}} On Dynamic Memory Allocation in Sliding-Window Parallel Patterns for Streaming Analytics
Massimo Torquati, Gabriele Mencagli, Maurizio Drocco, Marco Aldinucci, Tiziano De Matteis, Marco Danelutto
The Journal of Supercomputing
fastflow rephrase
This work studies the issues related to dynamic memory management in Data Stream Processing, an emerging paradigm enabling the real-time processing of live data streams. In this paper we consider two streaming parallel patterns and we discuss different implementation variants related on how dynamic memory is managed. The results show that the standard mechanisms provided by modern C++ are not entirely adequate for maximizing the performance. Instead, the combined use of an efficient general-purpose memory allocator, a custom allocator optimized for the pattern considered and a custom variant of the C++ shared pointer mechanism, provides a performance improvement up to 16% on the best case.
@article{17:dmadasp:jsupe,
abstract = {This work studies the issues related to dynamic memory management in Data Stream Processing, an emerging paradigm enabling the real-time processing of live data streams. In this paper we consider two streaming parallel patterns and we discuss different implementation variants related on how dynamic memory is managed. The results show that the standard mechanisms provided by modern C++ are not entirely adequate for maximizing the performance. Instead, the combined use of an efficient general-purpose memory allocator, a custom allocator optimized for the pattern considered and a custom variant of the C++ shared pointer mechanism, provides a performance improvement up to 16{\%} on the best case.},
author = {Torquati, Massimo and Mencagli, Gabriele and Drocco, Maurizio and Aldinucci, Marco and De Matteis, Tiziano and Danelutto, Marco},
doi = {10.1007/s11227-017-2152-1},
journal = {The Journal of Supercomputing},
keywords = {rephrase, fastflow},
number = {8},
pages = {4114--4131},
title = {On Dynamic Memory Allocation in Sliding-Window Parallel Patterns for Streaming Analytics},
url = {https://iris.unito.it/retrieve/handle/2318/1648626/362381/17_torquati_jsc.pdf},
volume = {75},
year = {2019},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1648626/362381/17_torquati_jsc.pdf},
bdsk-url-2 = {https://doi.org/10.1007/s11227-017-2152-1}} 2018
2018
PiCo: High-performance data analytics pipelines in modern C++
Claudia Misale, Maurizio Drocco, Guy Tremblay, Alberto R. Martinelli, Marco Aldinucci
Future Generation Computer Systems
fastflow HPC toreador
In this paper, we present a new C++ API with a fluent interface called PiCo (Pipeline Composition). PiCo's programming model aims at making easier the programming of data analytics applications while preserving or enhancing their performance. This is attained through three key design choices: (1) unifying batch and stream data access models, (2) decoupling processing from data layout, and (3) exploiting a stream-oriented, scalable, efficient C++11 runtime system. PiCo proposes a programming model based on pipelines and operators that are polymorphic with respect to data types in the sense that it is possible to reuse the same algorithms and pipelines on different data models (e.g., streams, lists, sets, etc.). Preliminary results show that PiCo, when compared to Spark and Flink, can attain better performances in terms of execution times and can hugely improve memory utilization, both for batch and stream processing.
@article{18:fgcs:pico,
abstract = {In this paper, we present a new C++ API with a fluent interface called PiCo (Pipeline Composition). PiCo's programming model aims at making easier the programming of data analytics applications while preserving or enhancing their performance. This is attained through three key design choices: (1) unifying batch and stream data access models, (2) decoupling processing from data layout, and (3) exploiting a stream-oriented, scalable, efficient C++11 runtime system. PiCo proposes a programming model based on pipelines and operators that are polymorphic with respect to data types in the sense that it is possible to reuse the same algorithms and pipelines on different data models (e.g., streams, lists, sets, etc.). Preliminary results show that PiCo, when compared to Spark and Flink, can attain better performances in terms of execution times and can hugely improve memory utilization, both for batch and stream processing.},
author = {Misale, Claudia and Drocco, Maurizio and Tremblay, Guy and Martinelli, Alberto R. and Aldinucci, Marco},
booktitle = {Future Generation Computer Systems},
date-added = {2018-05-18 21:24:31 +0000},
date-modified = {2020-11-15 17:22:30 +0100},
doi = {10.1016/j.future.2018.05.030},
journal = {Future Generation Computer Systems},
keywords = {toreador, HPC, fastflow},
pages = {392--403},
title = {PiCo: High-performance data analytics pipelines in modern C++},
url = {https://iris.unito.it/retrieve/handle/2318/1668444/414280/fgcs_pico.pdf},
volume = {87},
year = {2018},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1668444/414280/fgcs_pico.pdf},
bdsk-url-2 = {https://doi.org/10.1016/j.future.2018.05.030}} Harnessing sliding-window execution semantics for parallel stream processing
Gabriele Mencagli, Massimo Torquati, Fabio Lucattini, Salvatore Cuomo, Marco Aldinucci
Journal of Parallel and Distributed Computing
fastflow rephrase
Abstract According to the recent trend in data acquisition and processing technology, big data are increasingly available in the form of unbounded streams of elementary data items to be processed in real-time. In this paper we study in detail the paradigm of sliding windows, a well-known technique for approximated queries that update their results continuously as new fresh data arrive from the stream. In this work we focus on the relationship between the various existing sliding window semantics and the way the query processing is performed from the parallelism perspective. From this study two alternative parallel models are identified, each covering semantics with very precise properties. Each model is described in terms of its pros and cons, and parallel implementations in the FastFlow framework are analyzed by discussing the layout of the concurrent data structures used for the efficient windows representation in each model.
@article{17:slidingwindows:jpdc,
abstract = {Abstract According to the recent trend in data acquisition and processing technology, big data are increasingly available in the form of unbounded streams of elementary data items to be processed in real-time. In this paper we study in detail the paradigm of sliding windows, a well-known technique for approximated queries that update their results continuously as new fresh data arrive from the stream. In this work we focus on the relationship between the various existing sliding window semantics and the way the query processing is performed from the parallelism perspective. From this study two alternative parallel models are identified, each covering semantics with very precise properties. Each model is described in terms of its pros and cons, and parallel implementations in the FastFlow framework are analyzed by discussing the layout of the concurrent data structures used for the efficient windows representation in each model.},
author = {Gabriele Mencagli and Massimo Torquati and Fabio Lucattini and Salvatore Cuomo and Marco Aldinucci},
date-modified = {2018-12-27 18:24:34 +0100},
doi = {10.1016/j.jpdc.2017.10.021},
issn = {0743-7315},
journal = {Journal of Parallel and Distributed Computing},
keywords = {rephrase, fastflow},
month = jun,
pages = {74--88},
title = {Harnessing sliding-window execution semantics for parallel stream processing},
url = {https://iris.unito.it/retrieve/e27ce42c-1381-2581-e053-d805fe0acbaa/preprint-jpdc-2017.pdf},
volume = {116},
year = {2018},
bdsk-url-1 = {https://iris.unito.it/retrieve/e27ce42c-1381-2581-e053-d805fe0acbaa/preprint-jpdc-2017.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1016/j.jpdc.2017.10.021},
bdsk-url-3 = {https://iris.unito.it/retrieve/e27ce42c-1381-2581-e053-d805fe0acbaa/preprint-jpdc-2017.pdf}} 2017
2017
Parallel Programming with Global Asynchronous Memory: Models, C++ APIs and Implementations
Maurizio Drocco
Computer Science Department, University of Torino
fastflow paraphrase repara rephrase toreador
In the realm of High Performance Computing (HPC), message passing has been the programming paradigm of choice for over twenty years. The durable MPI (Message Passing Interface) standard, with send/receive communication, broadcast, gather/scatter, and reduction collectives is still used to construct parallel programs where each communication is orchestrated by the developer-based precise knowledge of data distribution and overheads; collective communications simplify the orchestration but might induce excessive synchronization. Early attempts to bring shared-memory programming model—with its programming advantages—to distributed computing, referred as the Distributed Shared Memory (DSM) model, faded away; one of the main issue was to combine performance and programmability with the memory consistency model. The recently proposed Partitioned Global Address Space (PGAS) model is a modern revamp of DSM that exposes data placement to enable optimizations based on locality, but it still addresses (simple) data-parallelism only and it relies on expensive sharing protocols. We advocate an alternative programming model for distributed computing based on a Global Asynchronous Memory (GAM), aiming to <i>avoid</i> coherency and consistency problems rather than solving them. We materialize GAM by designing and implementing a <i>distributed smart pointers</i> library, inspired by C++ smart pointers. In this model, public and private pointers (resembling C++ shared and unique pointers, respectively) are moved around instead of messages (i.e., data), thus alleviating the user from the burden of minimizing transfers. On top of smart pointers, we propose a high-level C++ template library for writing applications in terms of dataflow-like networks, namely GAM nets, consisting of stateful processors exchanging pointers in fully asynchronous fashion. We demonstrate the validity of the proposed approach, from the expressiveness perspective, by showing how GAM nets can be exploited to implement higher-level parallel programming models, such as data and task parallelism. As for the performance perspective, the execution of two non-toy benchmarks on a number of different small-scale HPC clusters exhibits both close-to-ideal scalability and negligible overhead with respect to state-of-the-art benchmark implementations. For instance, the GAM implementation of a high-quality video restoration filter sustains a 100 fps throughput over 70%-noisy high-quality video streams on a 4-node cluster of Graphics Processing Units (GPUs), with minimal programming effort.
@phdthesis{17:gam:drocco:thesis,
abstract = {In the realm of High Performance Computing (HPC), message passing has been the programming paradigm of choice for over twenty years. The durable MPI (Message Passing Interface) standard, with send/receive communication, broadcast, gather/scatter, and reduction collectives is still used to construct parallel programs where each communication is orchestrated by the de\-vel\-oper-based precise knowledge of data distribution and overheads; collective communications simplify the orchestration but might induce excessive synchronization. Early attempts to bring shared-memory programming model---with its programming adv\-antages---to distributed computing, referred as the Distributed Shared Memory (DSM) model, faded away; one of the main issue was to combine performance and programmability with the memory consistency model. The recently proposed Partitioned Global Address Space (PGAS) model is a modern revamp of DSM that exposes data placement to enable optimizations based on locality, but it still addresses (simple) data-parallelism only and it relies on expensive sharing protocols. We advocate an alternative programming model for distributed computing based on a Global Asynchronous Memory (GAM), aiming to \emph{avoid} coherency and consistency problems rather than solving them. We materialize GAM by designing and implementing a \emph{distributed smart pointers} library, inspired by C++ smart pointers. In this model, public and private pointers (resembling C++ shared and unique pointers, respectively) are moved around instead of messages (i.e., data), thus alleviating the user from the burden of minimizing transfers. On top of smart pointers, we propose a high-level C++ template library for writing applications in terms of dataflow-like networks, namely GAM nets, consisting of stateful processors exchanging pointers in fully asynchronous fashion. We demonstrate the validity of the proposed approach, from the expressiveness perspective, by showing how GAM nets can be exploited to implement higher-level parallel programming models, such as data and task parallelism. As for the performance perspective, the execution of two non-toy benchmarks on a number of different small-scale HPC clusters exhibits both close-to-ideal scalability and negligible overhead with respect to state-of-the-art benchmark implementations. For instance, the GAM implementation of a high-quality video restoration filter sustains a 100 fps throughput over 70\%-noisy high-quality video streams on a 4-node cluster of Graphics Processing Units (GPUs), with minimal programming effort.},
author = {Maurizio Drocco},
date-modified = {2017-12-12 15:09:35 +0000},
doi = {10.5281/zenodo.1037585},
keywords = {fastflow, rephrase, toreador, repara, paraphrase},
month = oct,
school = {Computer Science Department, University of Torino},
title = {Parallel Programming with Global Asynchronous Memory: Models, {C++} {API}s and Implementations},
url = {https://zenodo.org/record/1037585/files/Drocco_phd_thesis.pdf},
year = {2017},
bdsk-url-1 = {https://zenodo.org/record/1037585/files/Drocco_phd_thesis.pdf},
bdsk-url-2 = {http://dx.doi.org/10.5281/zenodo.1037585}} Towards Power-Aware Data Pipelining on Multicores
Marco Aldinucci, Marco Danelutto, Daniele De Sensi, Gabriele Mencagli, Massimo Torquati
Proceedings of the 10th International Symposium on High-Level Parallel Programming and Applications
fastflow rephrase
Power consumption management has become a major concern in software development. Continuous streaming computations are usually com- posed by different modules, exchanging data through shared message queues. The selection of the algorithm used to access such queues (i.e., the concurrency control) is a critical aspect for both performance and power consumption. In this paper, we describe the design of an adaptive concurrency control algo- rithm for implementing power-efficient communications on shared memory multicores. The algorithm provides the throughput offered by a nonblocking implementation and the power efficiency of a blocking protocol. We demon- strate that our algorithm reduces the power consumption of data streaming computations without decreasing their throughput.
@inproceedings{17:hlpp:powerstream,
abstract = {Power consumption management has become a major concern in software development. Continuous streaming computations are usually com- posed by different modules, exchanging data through shared message queues. The selection of the algorithm used to access such queues (i.e., the concurrency control) is a critical aspect for both performance and power consumption. In this paper, we describe the design of an adaptive concurrency control algo- rithm for implementing power-efficient communications on shared memory multicores. The algorithm provides the throughput offered by a nonblocking implementation and the power efficiency of a blocking protocol. We demon- strate that our algorithm reduces the power consumption of data streaming computations without decreasing their throughput.},
address = {Valladolid, Spain},
author = {Aldinucci, Marco and Danelutto, Marco and De Sensi, Daniele and Mencagli, Gabriele and Torquati, Massimo},
booktitle = {Proceedings of the 10th International Symposium on High-Level Parallel Programming and Applications},
date-added = {2017-07-13 09:02:32 +0000},
date-modified = {2018-12-27 18:53:27 +0100},
keywords = {rephrase, fastflow},
title = {Towards Power-Aware Data Pipelining on Multicores},
url = {https://iris.unito.it/retrieve/handle/2318/1644982/351415/17_HLPP_powerstream.pdf},
year = {2017},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1644982/351415/17_HLPP_powerstream.pdf}} PiCo: A Domain-Specific Language for Data Analytics Pipelines
Claudia Misale
Computer Science Department, University of Torino
fastflow paraphrase repara rephrase toreador
In the world of Big Data analytics, there is a series of tools aiming at simplifying programming applications to be executed on clusters. Although each tool claims to provide better programming, data and execution models—for which only informal (and often confusing) semantics is generally provided—all share a common under- lying model, namely, the Dataflow model. Using this model as a starting point, it is possible to categorize and analyze almost all aspects about Big Data analytics tools from a high level perspective. This analysis can be considered as a first step toward a formal model to be exploited in the design of a (new) framework for Big Data analytics. By putting clear separations between all levels of abstraction (i.e., from the runtime to the user API), it is easier for a programmer or software designer to avoid mixing low level with high level aspects, as we are often used to see in state-of-the-art Big Data analytics frameworks. From the user-level perspective, we think that a clearer and simple semantics is preferable, together with a strong separation of concerns. For this reason, we use the Dataflow model as a starting point to build a programming environment with a simplified programming model implemented as a Domain-Specific Language, that is on top of a stack of layers that build a prototypical framework for Big Data analytics. The contribution of this thesis is twofold: first, we show that the proposed model is (at least) as general as existing batch and streaming frameworks (e.g., Spark, Flink, Storm, Google Dataflow), thus making it easier to understand high-level data-processing applications written in such frameworks. As result of this analysis, we provide a layered model that can represent tools and applications following the Dataflow paradigm and we show how the analyzed tools fit in each level. Second, we propose a programming environment based on such layered model in the form of a Domain-Specific Language (DSL) for processing data collections, called PiCo (Pipeline Composition). The main entity of this programming model is the Pipeline, basically a DAG-composition of processing elements. This model is intended to give the user an unique interface for both stream and batch processing, hiding completely data management and focusing only on operations, which are represented by Pipeline stages. Our DSL will be built on top of the FastFlow library, exploiting both shared and distributed parallelism, and implemented in C++11/14 with the aim of porting C++ into the Big Data world.
@phdthesis{17:pico:misale:thesis,
abstract = {
In the world of Big Data analytics, there is a series of tools aiming at simplifying programming applications to be executed on clusters. Although each tool claims to provide better programming, data and execution models---for which only informal (and often confusing) semantics is generally provided---all share a common under- lying model, namely, the Dataflow model. Using this model as a starting point, it is possible to categorize and analyze almost all aspects about Big Data analytics tools from a high level perspective. This analysis can be considered as a first step toward a formal model to be exploited in the design of a (new) framework for Big Data analytics. By putting clear separations between all levels of abstraction (i.e., from the runtime to the user API), it is easier for a programmer or software designer to avoid mixing low level with high level aspects, as we are often used to see in state-of-the-art Big Data analytics frameworks.
From the user-level perspective, we think that a clearer and simple semantics is preferable, together with a strong separation of concerns. For this reason, we use the Dataflow model as a starting point to build a programming environment with a simplified programming model implemented as a Domain-Specific Language, that is on top of a stack of layers that build a prototypical framework for Big Data analytics.
The contribution of this thesis is twofold: first, we show that the proposed model is (at least) as general as existing batch and streaming frameworks (e.g., Spark, Flink, Storm, Google Dataflow), thus making it easier to understand high-level data-processing applications written in such frameworks. As result of this analysis, we provide a layered model that can represent tools and applications following the Dataflow paradigm and we show how the analyzed tools fit in each level.
Second, we propose a programming environment based on such layered model in the form of a Domain-Specific Language (DSL) for processing data collections, called PiCo (Pipeline Composition). The main entity of this programming model is the Pipeline, basically a DAG-composition of processing elements. This model is intended to give the user an unique interface for both stream and batch processing, hiding completely data management and focusing only on operations, which are represented by Pipeline stages. Our DSL will be built on top of the FastFlow library, exploiting both shared and distributed parallelism, and implemented in C++11/14 with the aim of porting C++ into the Big Data world.
},
author = {Claudia Misale},
date-added = {2017-06-19 15:15:52 +0000},
date-modified = {2017-06-19 15:55:21 +0000},
doi = {10.5281/zenodo.579753},
keywords = {fastflow, rephrase, toreador, repara, paraphrase},
month = may,
school = {Computer Science Department, University of Torino},
title = {PiCo: A Domain-Specific Language for Data Analytics Pipelines},
url = {https://iris.unito.it/retrieve/handle/2318/1633743/320170/Misale_thesis.pdf},
year = {2017},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1633743/320170/Misale_thesis.pdf},
bdsk-url-2 = {http://dx.doi.org/10.5281/zenodo.579753}} FastFlow: high-level and efficient streaming on multi-core
Marco Aldinucci, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
Programming Multi-core and Many-core Computing Systems
fastflow
This chapter first outlines FastFlow design and then shows sample use of the FastFlow programming environment together with performance results achieved on various state-of-the-art multicore architectures. The FastFlow framework has been designed according to four foundational principles: layered design; efficiency in base mechanisms; support for stream parallelism; and a programming model based on design pattern/algorithmic skeleton concepts. The core of the FastFlow framework provides an efficient implementation of single-producer-single-consumer (SPSC) first in-first out (FIFO) queues. The next tier up extends from one-to-one queues to one-to-many, many-to-one, and many-to-many synchronizations and data flows, which are implemented using only SPSC queues and arbiter threads, thus providing lock-free and wait-free arbitrary dataflow graphs. When designing and implementing new parallel applications using FastFlow, programmers instantiate patterns provided by FastFlow to adapt them to the specific needs of the application at hand. The chapter demonstrates how the principal FastFlow patterns may be used in a parallel application.
@inbook{ff:wileybook:17,
abstract = {This chapter first outlines FastFlow design and then shows sample use of the FastFlow programming environment together with performance results achieved on various state-of-the-art multicore architectures. The FastFlow framework has been designed according to four foundational principles: layered design; efficiency in base mechanisms; support for stream parallelism; and a programming model based on design pattern/algorithmic skeleton concepts. The core of the FastFlow framework provides an efficient implementation of single-producer-single-consumer (SPSC) first in-first out (FIFO) queues. The next tier up extends from one-to-one queues to one-to-many, many-to-one, and many-to-many synchronizations and data flows, which are implemented using only SPSC queues and arbiter threads, thus providing lock-free and wait-free arbitrary dataflow graphs. When designing and implementing new parallel applications using FastFlow, programmers instantiate patterns provided by FastFlow to adapt them to the specific needs of the application at hand. The chapter demonstrates how the principal FastFlow patterns may be used in a parallel application.},
author = {Marco Aldinucci and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
booktitle = {Programming Multi-core and Many-core Computing Systems},
chapter = {13},
date-added = {2011-06-18 18:28:00 +0200},
date-modified = {2014-12-31 14:14:28 +0000},
doi = {10.1002/9781119332015.ch13},
editor = {Sabri Pllana and Fatos Xhafa},
isbn = {9781119332015},
keywords = {fastflow},
optannote = {ISBN: 0470936908},
pages = {261--280},
publisher = {John Wiley \& Sons, Ltd},
series = {Parallel and Distributed Computing},
title = {FastFlow: high-level and efficient streaming on multi-core},
url = {http://calvados.di.unipi.it/storage/paper_files/2011_FF_tutorial-draft.pdf},
year = {2017},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2011_FF_tutorial-draft.pdf},
bdsk-url-2 = {https://dx.doi.org/10.1002/9781119332015.ch13}} NuChart-II: the road to a fast and scalable tool for Hi-C data analysis
Fabio Tordini, Maurizio Drocco, Claudia Misale, Luciano Milanesi, Pietro Liò, Ivan Merelli, Massimo Torquati, Marco Aldinucci
International Journal of High Performance Computing Applications
bioinformatics fastflow repara rephrase
Recent advances in molecular biology and bioinformatics techniques brought to an explosion of the information about the spatial organisation of the DNA in the nucleus of a cell. High-throughput molecular biology techniques provide a genome-wide capture of the spatial organization of chromosomes at unprecedented scales, which permit to identify physical interactions between genetic elements located throughout a genome. Recent results have shown that there is a large correlation between co-localization and co-regulation of genes, but these important information are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we present NuChart-II, an efficient and highly optimized tool for genomic data analysis that provides a gene-centric, graph-based representation of genomic information. While designing NuChart-II we addressed several common issues in the parallelisation of memory bound algorithms for shared-memory systems. With performance and usability in mind, NuChart-II is a R package that embeds a C++ engine: computing capabilities and memory hierarchy of multi-core architectures are fully exploited, while the versatile R environment for statistical analysis and data visualisation rises the level of abstraction and permits to orchestrate analysis and visualisation of genomic data.
@article{16:ijhpca:nuchart,
abstract = {Recent advances in molecular biology and bioinformatics techniques brought to an explosion of the information about the spatial organisation of the DNA in the nucleus of a cell. High-throughput molecular biology techniques provide a genome-wide capture of the spatial organization of chromosomes at unprecedented scales, which permit to identify physical interactions between genetic elements located throughout a genome. Recent results have shown that there is a large correlation between co-localization and co-regulation of genes, but these important information are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we present NuChart-II, an efficient and highly optimized tool for genomic data analysis that provides a gene-centric, graph-based representation of genomic information. While designing NuChart-II we addressed several common issues in the parallelisation of memory bound algorithms for shared-memory systems. With performance and usability in mind, NuChart-II is a R package that embeds a C++ engine: computing capabilities and memory hierarchy of multi-core architectures are fully exploited, while the versatile R environment for statistical analysis and data visualisation rises the level of abstraction and permits to orchestrate analysis and visualisation of genomic data.},
author = {Fabio Tordini and Maurizio Drocco and Claudia Misale and Luciano Milanesi and Pietro Li{\`o} and Ivan Merelli and Massimo Torquati and Marco Aldinucci},
date-modified = {2018-12-27 19:06:22 +0100},
doi = {10.1177/1094342016668567},
journal = {International Journal of High Performance Computing Applications},
keywords = {fastflow, bioinformatics, repara, rephrase, bioinformatics},
number = {3},
pages = {196--211},
title = {{NuChart-II}: the road to a fast and scalable tool for {Hi-C} data analysis},
url = {https://iris.unito.it/retrieve/handle/2318/1607126/238747/main.pdf},
volume = {31},
year = {2017},
bdsk-url-1 = {http://hdl.handle.net/2318/1607126},
bdsk-url-2 = {http://dx.doi.org/10.1177/1094342016668567},
bdsk-url-3 = {https://iris.unito.it/retrieve/handle/2318/1607126/238747/main.pdf}} 2016
2016
A Cluster-As-Accelerator approach for SPMD-free Data Parallelism
Maurizio Drocco, Claudia Misale, Marco Aldinucci
Proc. of 24th Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)
fastflow rephrase
In this paper we present a novel approach for functional-style programming of distributed-memory clusters, targeting data-centric applications. The programming model proposed is purely sequential, SPMD-free and based on high- level functional features introduced since C++11 specification. Additionally, we propose a novel cluster-as-accelerator design principle. In this scheme, cluster nodes act as general inter- preters of user-defined functional tasks over node-local portions of distributed data structures. We envision coupling a simple yet powerful programming model with a lightweight, locality- aware distributed runtime as a promising step along the road towards high-performance data analytics, in particular under the perspective of the upcoming exascale era. We implemented the proposed approach in SkeDaTo, a prototyping C++ library of data-parallel skeletons exploiting cluster-as-accelerator at the bottom layer of the runtime software stack.
@inproceedings{skedato:pdp:16,
abstract = {In this paper we present a novel approach for functional-style programming of distributed-memory clusters, targeting data-centric applications. The programming model proposed is purely sequential, SPMD-free and based on high- level functional features introduced since C++11 specification. Additionally, we propose a novel cluster-as-accelerator design principle. In this scheme, cluster nodes act as general inter- preters of user-defined functional tasks over node-local portions of distributed data structures. We envision coupling a simple yet powerful programming model with a lightweight, locality- aware distributed runtime as a promising step along the road towards high-performance data analytics, in particular under the perspective of the upcoming exascale era. We implemented the proposed approach in SkeDaTo, a prototyping C++ library of data-parallel skeletons exploiting cluster-as-accelerator at the bottom layer of the runtime software stack.},
address = {Crete, Greece},
author = {Maurizio Drocco and Claudia Misale and Marco Aldinucci},
booktitle = {Proc. of 24th Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)},
date-modified = {2017-12-12 14:49:31 +0000},
doi = {10.1109/PDP.2016.97},
keywords = {rephrase, fastflow},
pages = {350--353},
publisher = {IEEE},
title = {A Cluster-As-Accelerator approach for {SPMD}-free Data Parallelism},
url = {https://iris.unito.it/retrieve/handle/2318/1611858/262689/2016_pdp_skedato.pdf},
year = {2016},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1611858/262689/2016_pdp_skedato.pdf},
bdsk-url-2 = {https://doi.org/10.1109/PDP.2016.97}} RPL: A Domain-Specific Language for Designing and Implementing Parallel C++ Applications
Vladimir Janjic, Christopher Brown, Kenneth MacKenzie, Kevin Hammond, Marco Danelutto, Marco Aldinucci, José Daniel Garcı́a
Proc. of Intl. Euromicro PDP 2016: Parallel Distributed and network-based Processing
fastflow rephrase
Parallelising sequential applications is usually a very hard job, due to many different ways in which an application can be parallelised and a large number of programming models (each with its own advantages and disadvantages) that can be used. In this paper, we describe a method to semi- automatically generate and evaluate different parallelisations of the same application, allowing programmers to find the best parallelisation without significant manual reengineering of the code. We describe a novel, high-level domain-specific language, Refactoring Pattern Language (RPL), that is used to represent the parallel structure of an application and to capture its extra-functional properties (such as service time). We then describe a set of RPL rewrite rules that can be used to generate alternative, but semantically equivalent, parallel structures (parallelisations) of the same application. We also describe the RPL Shell that can be used to evaluate these parallelisations, in terms of the desired extra-functional properties. Finally, we describe a set of C++ refactorings, targeting OpenMP, Intel TBB and FastFlow parallel programming models, that semi-automatically apply the desired parallelisation to the application's source code, therefore giving a parallel version of the code. We demonstrate how the RPL and the refactoring rules can be used to derive efficient parallelisations of two realistic C++ use cases (Image Convolution and Ant Colony Optimisation).
@inproceedings{rpl:pdp:16,
abstract = {Parallelising sequential applications is usually a very hard job, due to many different ways in which an application can be parallelised and a large number of programming models (each with its own advantages and disadvantages) that can be used. In this paper, we describe a method to semi- automatically generate and evaluate different parallelisations of the same application, allowing programmers to find the best parallelisation without significant manual reengineering of the code. We describe a novel, high-level domain-specific language, Refactoring Pattern Language (RPL), that is used to represent the parallel structure of an application and to capture its extra-functional properties (such as service time). We then describe a set of RPL rewrite rules that can be used to generate alternative, but semantically equivalent, parallel structures (parallelisations) of the same application. We also describe the RPL Shell that can be used to evaluate these parallelisations, in terms of the desired extra-functional properties. Finally, we describe a set of C++ refactorings, targeting OpenMP, Intel TBB and FastFlow parallel programming models, that semi-automatically apply the desired parallelisation to the application's source code, therefore giving a parallel version of the code. We demonstrate how the RPL and the refactoring rules can be used to derive efficient parallelisations of two realistic C++ use cases (Image Convolution and Ant Colony Optimisation).},
address = {Crete, Greece},
author = {Janjic, Vladimir and Brown, Christopher and MacKenzie, Kenneth and Hammond, Kevin and Danelutto, Marco and Aldinucci, Marco and Garc{\'{\i}}a, Jos{\'{e}} Daniel},
booktitle = {Proc. of Intl. Euromicro PDP 2016: Parallel Distributed and network-based Processing},
date-modified = {2017-06-20 08:19:39 +0000},
doi = {10.1109/PDP.2016.122},
keywords = {rephrase, fastflow},
publisher = {IEEE},
title = {{RPL}: A Domain-Specific Language for Designing and Implementing Parallel C++ Applications},
url = {https://iris.unito.it/retrieve/handle/2318/1597172/299237/2016_jsupe_stencil_pp_4aperto.pdf},
year = {2016},
bdsk-url-1 = {http://hdl.handle.net/2318/1597172},
bdsk-url-2 = {https://iris.unito.it/retrieve/handle/2318/1597172/299237/2016_jsupe_stencil_pp_4aperto.pdf},
bdsk-url-3 = {http://dx.doi.org/10.1109/PDP.2016.122}} Pool Evolution: A Parallel Pattern for Evolutionary and Symbolic Computing
Marco Aldinucci, Sonia Campa, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
International Journal of Parallel Programming
fastflow paraphrase repara
We introduce a new parallel pattern derived from a specific application domain and show how it turns out to have application beyond its domain of origin. The pool evolution pattern models the parallel evolution of a population subject to mutations and evolving in such a way that a given fitness function is optimized. The pattern has been demonstrated to be suitable for capturing and modeling the parallel patterns underpinning various evolutionary algorithms, as well as other parallel patterns typical of symbolic computation. In this paper we introduce the pattern, we discuss its implementation on modern multi/many core architectures and finally present experimental results obtained with FastFlow and Erlang implementations to assess its feasibility and scalability.
@article{pool:ijpp:15,
abstract = {We introduce a new parallel pattern derived from a specific application domain and show how it turns out to have application beyond its domain of origin. The pool evolution pattern models the parallel evolution of a population subject to mutations and evolving in such a way that a given fitness function is optimized. The pattern has been demonstrated to be suitable for capturing and modeling the parallel patterns underpinning various evolutionary algorithms, as well as other parallel patterns typical of symbolic computation. In this paper we introduce the pattern, we discuss its implementation on modern multi/many core architectures and finally present experimental results obtained with FastFlow and Erlang implementations to assess its feasibility and scalability.},
author = {Marco Aldinucci and Sonia Campa and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
date-added = {2015-03-21 22:15:47 +0000},
date-modified = {2015-09-24 11:15:53 +0000},
doi = {10.1007/s10766-015-0358-5},
issn = {0885-7458},
journal = {International Journal of Parallel Programming},
keywords = {fastflow, paraphrase, repara},
number = {3},
pages = {531--551},
publisher = {Springer US},
title = {Pool Evolution: A Parallel Pattern for Evolutionary and Symbolic Computing},
url = {https://iris.unito.it/retrieve/handle/2318/1522392/42139/2015_ff_pool_ijpp.pdf},
volume = {44},
year = {2016},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1522392/42139/2015_ff_pool_ijpp.pdf},
bdsk-url-2 = {https://doi.org/10.1007/s10766-015-0358-5}} NuchaRt: embedding high-level parallel computing in R for augmented Hi-C data analysis
Fabio Tordini, Ivan Merelli, Pietro Liò, Luciano Milanesi, Marco Aldinucci
Computational Intelligence Methods for Bioinformatics and Biostatistics
bioinformatics fastflow repara
Recent advances in molecular biology and Bioinformatics techniques brought to an explosion of the information about the spatial organisation of the DNA in the nucleus. High-throughput chromosome conformation capture techniques provide a genome-wide capture of chromatin contacts at unprecedented scales, which permit to identify physical interactions between genetic elements located throughout the human genome. These important studies are hampered by the lack of biologists-friendly software. In this work we present NuchaRt, an R package that wraps NuChart-II, an efficient and highly optimized C++ tool for the exploration of Hi-C data. By rising the level of abstraction, NuchaRt proposes a high-performance pipeline that allows users to orchestrate analysis and visualisation of multi-omics data, making optimal use of the computing capabilities offered by modern multi-core architectures, combined with the versatile and well known R environment for statistical analysis and data visualisation.
@incollection{15:lnbi:nuchaRt,
abstract = {Recent advances in molecular biology and Bioinformatics techniques brought to an explosion of the information about the spatial organisation of the DNA in the nucleus. High-throughput chromosome conformation capture techniques provide a genome-wide capture of chromatin contacts at unprecedented scales, which permit to identify physical interactions between genetic elements located throughout the human genome. These important studies are hampered by the lack of biologists-friendly software. In this work we present NuchaRt, an R package that wraps NuChart-II, an efficient and highly optimized C++ tool for the exploration of Hi-C data. By rising the level of abstraction, NuchaRt proposes a high-performance pipeline that allows users to orchestrate analysis and visualisation of multi-omics data, making optimal use of the computing capabilities offered by modern multi-core architectures, combined with the versatile and well known R environment for statistical analysis and data visualisation.},
address = {Cham (ZG)},
author = {Fabio Tordini and Ivan Merelli and Pietro Li{\`o} and Luciano Milanesi and Marco Aldinucci},
booktitle = {Computational Intelligence Methods for Bioinformatics and Biostatistics},
doi = {10.1007/978-3-319-44332-4},
editor = {Springer International Publishing},
isbn = {978-3-319-44331-7},
keywords = {fastflow, bioinformatics, repara, bioinformatics},
pages = {259--272},
publisher = {Springer International Publishing},
series = {{Lecture Notes in Computer Science}},
title = {{NuchaRt}: embedding high-level parallel computing in {R} for augmented {Hi-C} data analysis},
url = {https://iris.unito.it/retrieve/handle/2318/1608281/253372/rnuchart.pdf},
volume = {9874},
year = {2016},
bdsk-url-1 = {http://link.springer.com/book/10.1007%2F978-3-319-44332-4},
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-319-44332-4}} A cloud solution for multi-omics data integration
Fabio Tordini
Proceedings of the 16th IEEE International Conference on Scalable Computing and Communication
bioinformatics fastflow rephrase
Recent advances in molecular biology and Bioinformatics techniques have brought to an explosion of the information about the spatial organisation of the DNA inside the nucleus. In particular, 3C-based techniques are revealing the genome folding for many different cell types, and permit to create a more effective representation of the disposition of genes in the three-dimensional space. This information can be used to re-interpret heterogeneous genomic data (multi-omic) relying on 3D maps of the chromosome. The storage and computational requirements needed to accomplish such operations on raw sequenced data have to be fulfilled using HPC solutions, and the the Cloud paradigm is a valuable and convenient mean for delivering HPC to Bioinformatics. In this work we describe a data analysis work-flow that allows the integration and the interpretation of multi-omic data on a sort of “topographical” nuclear map, capable of representing the effective disposition of genes in a graph-based representation. We propose a cloud-based task farm pattern to orchestrate the services needed to accomplish genomic data analysis, where each service represents a special-purpose tool, playing a part in well known data analysis pipelines.
@inproceedings{16:scalcom:cloud,
abstract = {Recent advances in molecular biology and Bioinformatics techniques have brought to an explosion of the information about the spatial organisation of the DNA inside the nucleus. In particular, 3C-based techniques are revealing the genome folding for many different cell types, and permit to create a more effective representation of the disposition of genes in the three-dimensional space. This information can be used to re-interpret heterogeneous genomic data (multi-omic) relying on 3D maps of the chromosome. The storage and computational requirements needed to accomplish such operations on raw sequenced data have to be fulfilled using HPC solutions, and the the Cloud paradigm is a valuable and convenient mean for delivering HPC to Bioinformatics. In this work we describe a data analysis work-flow that allows the integration and the interpretation of multi-omic data on a sort of ``topographical'' nuclear map, capable of representing the effective disposition of genes in a graph-based representation. We propose a cloud-based task farm pattern to orchestrate the services needed to accomplish genomic data analysis, where each service represents a special-purpose tool, playing a part in well known data analysis pipelines.},
author = {Fabio Tordini},
booktitle = {Proceedings of the 16th IEEE International Conference on Scalable Computing and Communication},
date-modified = {2016-08-30 10:26:12 +0000},
doi = {10.1109/UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.131},
keywords = {fastflow, bioinformatics, rephrase},
note = {Best paper award},
pages = {559--566},
publisher = {IEEE Computer Society},
title = {{A cloud solution for multi-omics data integration}},
url = {http://calvados.di.unipi.it/storage/paper_files/2016_cloudpipeline_scalcom.pdf},
year = {2016},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2016_cloudpipeline_scalcom.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1109/UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.131}} The road towards a Cloud-based High-Performance solution for genomic data analysis
Fabio Tordini
Computer Science Department, University of Torino, Italy
bioinformatics fastflow
Nowadays, molecular biology laboratories are delivering more and more data about DNA organisation, at increasing resolution and in a large number of samples. So much that genomic research is now facing many of the scale-out issues that high-performance computing has been addressing for years: they require powerful infrastructures with fast computing and storage capabilities, with substantial challenges in terms of data processing, statistical analysis and data representation. With this thesis we propose a high-performance pipeline for the analysis and interpretation of heterogeneous genomic information: beside performance, usability and availability are two essential requirements that novel Bioinformatics tools should satisfy. In this perspective, we propose and discuss our efforts towards a solid infrastructure for data processing and storage, where software that operates over data is exposed as a service, and is accessible by users through the Internet. We begin by presenting NuChart-II, a tool for the analysis and interpretation of spatial genomic information. With NuChart-II we propose a graph-based representation of genomic data, which can provide insights on the disposition of genomic elements in the DNA. We also discuss our approach for the normalisation of biases that affect raw sequenced data. We believe that many currently available tools for genomic data analysis are perceived as tricky and troublesome applications, that require highly specialised skills to obtain the desired outcomes. Concerning usability, we want to rise the level of abstraction perceived by the user, but maintain high performance and correctness while providing an exhaustive solution for data visualisation. We also intend to foster the availability of novel tools: in this work we also discuss a cloud solution that delivers computation and storage as dynamically allocated virtual resources via the Internet, while needed software is provided as a service. In this way, the computational demand of genomic research can be satisfied more economically by using lab-scale and enterprise-oriented technologies. Here we discuss our idea of a task farm for the integration of heterogeneous data resulting from different sequencing experiments: we believe that the integration of multi-omic features on a nuclear map can be a valuable mean for studying the interactions among genetic elements. This can reveal insights on biological mechanisms, such as genes regulation, translocations and epigenetic patterns.
@phdthesis{tordiniThesis16,
abstract = {Nowadays, molecular biology laboratories are delivering more and more data about DNA organisation, at increasing resolution and in a large number of samples. So much that genomic research is now facing many of the scale-out issues that high-performance computing has been addressing for years: they require powerful infrastructures with fast computing and storage capabilities, with substantial challenges in terms of data processing, statistical analysis and data representation. With this thesis we propose a high-performance pipeline for the analysis and interpretation of heterogeneous genomic information: beside performance, usability and availability are two essential requirements that novel Bioinformatics tools should satisfy. In this perspective, we propose and discuss our efforts towards a solid infrastructure for data processing and storage, where software that operates over data is exposed as a service, and is accessible by users through the Internet. We begin by presenting NuChart-II, a tool for the analysis and interpretation of spatial genomic information. With NuChart-II we propose a graph-based representation of genomic data, which can provide insights on the disposition of genomic elements in the DNA. We also discuss our approach for the normalisation of biases that affect raw sequenced data. We believe that many currently available tools for genomic data analysis are perceived as tricky and troublesome applications, that require highly specialised skills to obtain the desired outcomes. Concerning usability, we want to rise the level of abstraction perceived by the user, but maintain high performance and correctness while providing an exhaustive solution for data visualisation. We also intend to foster the availability of novel tools: in this work we also discuss a cloud solution that delivers computation and storage as dynamically allocated virtual resources via the Internet, while needed software is provided as a service. In this way, the computational demand of genomic research can be satisfied more economically by using lab-scale and enterprise-oriented technologies. Here we discuss our idea of a task farm for the integration of heterogeneous data resulting from different sequencing experiments: we believe that the integration of multi-omic features on a nuclear map can be a valuable mean for studying the interactions among genetic elements. This can reveal insights on biological mechanisms, such as genes regulation, translocations and epigenetic patterns.},
author = {Fabio Tordini},
keywords = {fastflow, bioinformatics},
month = apr,
school = {Computer Science Department, University of Torino, Italy},
title = {{The road towards a Cloud-based High-Performance solution for genomic data analysis}},
url = {http://calvados.di.unipi.it/storage/paper_files/2016_tordini_phdthesis.pdf},
year = {2016},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2016_tordini_phdthesis.pdf}} Embedding Semantics of the Single-Producer/Single-Consumer Lock-Free Queue into a Race Detection Tool
Manuel F. Dolz, David Rio Astorga, Javier Fernández, José Daniel Garcı́a, Félix Garcı́a Carballeira, Marco Danelutto, Massimo Torquati
Proceedings of the 7th International Workshop on Programming Models and Applications for Multicores and Manycores
fastflow repara
@inproceedings{16:PMAM:SPSC,
acmid = {2883406},
address = {New York, NY, USA},
author = {Dolz, Manuel F. and del Rio Astorga, David and Fern{\'{a}}ndez, Javier and Garc{\'{\i}}a, Jos{\'{e}} Daniel and Garc{\'{\i}}a Carballeira, F{\'{e}}lix and Danelutto, Marco and Torquati, Massimo},
booktitle = {Proceedings of the 7th International Workshop on Programming Models and Applications for Multicores and Manycores},
date-modified = {2016-04-21 17:33:36 +0000},
doi = {10.1145/2883404.2883406},
isbn = {978-1-4503-4196-7},
keywords = {fastflow, repara},
location = {Barcelona, Spain},
numpages = {10},
pages = {20--29},
publisher = {ACM},
series = {PMAM'16},
title = {Embedding Semantics of the Single-Producer/Single-Consumer Lock-Free Queue into a Race Detection Tool},
url = {https://doi.acm.org/10.1145/2883404.2883406},
year = {2016},
bdsk-url-1 = {https://doi.acm.org/10.1145/2883404.2883406},
bdsk-url-2 = {http://dx.doi.org/10.1145/2883404.2883406}} pWhatsHap: efficient haplotyping for future generation sequencing
Andrea Bracciali, Marco Aldinucci, Murray Patterson, Tobias Marschall, Nadia Pisanti, Ivan Merelli, Massimo Torquati
BMC Bioinformatics
fastflow paraphrase rephrase
Background: Haplotype phasing is an important problem in the analysis of genomics information. Given a set of DNA fragments of an individual, it consists of determining which one of the possible alleles (alternative forms of a gene) each fragment comes from. Haplotype information is relevant to gene regulation, epigenetics, genome-wide association studies, evolutionary and population studies, and the study of mutations. Haplotyping is currently addressed as an optimisation problem aiming at solutions that minimise, for instance, error correction costs, where costs are a measure of the confidence in the accuracy of the information acquired from DNA sequencing. Solutions have typically an exponential computational complexity. WhatsHap is a recent optimal approach which moves computational complexity from DNA fragment length to fragment overlap, i.e., coverage, and is hence of particular interest when considering sequencing technology's current trends that are producing longer fragments. Results: Given the potential relevance of efficient haplotyping in several analysis pipelines, we have designed and engineered pWhatsHap, a parallel, high-performance version of WhatsHap. pWhatsHap is embedded in a toolkit developed in Python and supports genomics datasets in standard file formats. Building on WhatsHap, pWhatsHap exhibits the same complexity exploring a number of possible solutions which is exponential in the coverage of the dataset. The parallel implementation on multi-core architectures allows for a relevant reduction of the execution time for haplotyping, while the provided results enjoy the same high accuracy as that provided by WhatsHap, which increases with coverage. Conclusions: Due to its structure and management of the large datasets, the parallelisation of WhatsHap posed demanding technical challenges, which have been addressed exploiting a high-level parallel programming framework. The result, pWhatsHap, is a freely available toolkit that improves the efficiency of the analysis of genomics information.
@article{16:pwhatshap:bmc,
abstract = {Background: Haplotype phasing is an important problem in the analysis of genomics information. Given a set of DNA fragments of an individual, it consists of determining which one of the possible alleles (alternative forms of a gene) each fragment comes from. Haplotype information is relevant to gene regulation, epigenetics, genome-wide association studies, evolutionary and population studies, and the study of mutations. Haplotyping is currently addressed as an optimisation problem aiming at solutions that minimise, for instance, error correction costs, where costs are a measure of the confidence in the accuracy of the information acquired from DNA sequencing. Solutions have typically an exponential computational complexity. WhatsHap is a recent optimal approach which moves computational complexity from DNA fragment length to fragment overlap, i.e., coverage, and is hence of particular interest when considering sequencing technology's current trends that are producing longer fragments. Results: Given the potential relevance of efficient haplotyping in several analysis pipelines, we have designed and engineered pWhatsHap, a parallel, high-performance version of WhatsHap. pWhatsHap is embedded in a toolkit developed in Python and supports genomics datasets in standard file formats. Building on WhatsHap, pWhatsHap exhibits the same complexity exploring a number of possible solutions which is exponential in the coverage of the dataset. The parallel implementation on multi-core architectures allows for a relevant reduction of the execution time for haplotyping, while the provided results enjoy the same high accuracy as that provided by WhatsHap, which increases with coverage. Conclusions: Due to its structure and management of the large datasets, the parallelisation of WhatsHap posed demanding technical challenges, which have been addressed exploiting a high-level parallel programming framework. The result, pWhatsHap, is a freely available toolkit that improves the efficiency of the analysis of genomics information.},
author = {Andrea Bracciali and Marco Aldinucci and Murray Patterson and Tobias Marschall and Nadia Pisanti and Ivan Merelli and Massimo Torquati},
date-modified = {2016-10-17 17:28:27 +0000},
doi = {10.1186/s12859-016-1170-y},
journal = {BMC Bioinformatics},
keywords = {fastflow, paraphrase, rephrase},
number = {Suppl 11},
pages = {342},
title = {pWhatsHap: efficient haplotyping for future generation sequencing},
url = {http://bmcbioinformatics.biomedcentral.com/track/pdf/10.1186/s12859-016-1170-y?site=bmcbioinformatics.biomedcentral.com},
volume = {17},
year = {2016},
bdsk-url-1 = {http://hdl.handle.net/2318/1607125},
bdsk-url-2 = {http://bmcbioinformatics.biomedcentral.com/track/pdf/10.1186/s12859-016-1170-y?site=bmcbioinformatics.biomedcentral.com},
bdsk-url-3 = {http://dx.doi.org/10.1186/s12859-016-1170-y}} 2015
2015
Progettazione e sviluppo di un provider libfabric per la rete ad alte prestazioni Ronniee/A3Cube
Paolo Inaudi
Computer Science Department, University of Torino
fastflow
@mastersthesis{tesi:inaudi:15,
author = {Paolo Inaudi},
keywords = {fastflow},
month = oct,
school = {Computer Science Department, University of Torino},
title = {Progettazione e sviluppo di un provider libfabric per la rete ad alte prestazioni Ronniee/A3Cube},
year = {2015}} Parallel Computing Techniques for High Energy Physics
Paolo Viviani
Physics Department, University of Torino
fastflow impact
Modern experimental achievements, with LHC results as a prominent but not exclusive representative, have undisclosed a new range of challenges concerning theoretical com- putations. Tree level QED calculation are no more satisfactory due to the very small experimental uncertainty of precision e+ e- measurements, so Next To Leading and Next to Next to Leading Order calculations are required. At the same time many-legs, high-order QCD processes needed to simulate LHC events are raising even more the bar of computational complexity. The drive for the present work has been the interest in calculating high multiplicity Higgs boson processes with a dedicated software library (RECOLA) currently under development at the University of Torino, as well as the related technological challenges. This thesis undertakes the task of exploring the possibilities offered by present and upcoming computing technologies in order to face these challenges properly. The first two chapters outlines the theoretical context and the available technologies. In chapter 3 a a case study is examined in full detail, in order to explore the suitability of different parallel computing solutions. In the chapter 4, some of those solutions are implemented in the context of the RECOLA library, allowing it to handle processes at a previously unexplored scale of complexity. Alongside, the potential of new, cost-effective parallel architectures is tested.
@mastersthesis{tesi:viviani:15,
abstract = {Modern experimental achievements, with LHC results as a prominent but not exclusive representative, have undisclosed a new range of challenges concerning theoretical com- putations. Tree level QED calculation are no more satisfactory due to the very small experimental uncertainty of precision e+ e- measurements, so Next To Leading and Next to Next to Leading Order calculations are required. At the same time many-legs, high-order QCD processes needed to simulate LHC events are raising even more the bar of computational complexity. The drive for the present work has been the interest in calculating high multiplicity Higgs boson processes with a dedicated software library (RECOLA) currently under development at the University of Torino, as well as the related technological challenges. This thesis undertakes the task of exploring the possibilities offered by present and upcoming computing technologies in order to face these challenges properly. The first two chapters outlines the theoretical context and the available technologies. In chapter 3 a a case study is examined in full detail, in order to explore the suitability of different parallel computing solutions. In the chapter 4, some of those solutions are implemented in the context of the RECOLA library, allowing it to handle processes at a previously unexplored scale of complexity. Alongside, the potential of new, cost-effective parallel architectures is tested.},
author = {Paolo Viviani},
date-added = {2015-09-27 12:36:54 +0000},
date-modified = {2015-09-27 13:28:24 +0000},
keywords = {fastflow,impact},
school = {Physics Department, University of Torino},
title = {Parallel Computing Techniques for High Energy Physics},
year = {2015}} The Loop-of-Stencil-Reduce paradigm
Marco Aldinucci, Marco Danelutto, Maurizio Drocco, Peter Kilpatrick, Guilherme Peretti Pezzi, Massimo Torquati
Proc. of Intl. Workshop on Reengineering for Parallelism in Heterogeneous Parallel Platforms (RePara)
fastflow HPC repara
In this paper we advocate the Loop-of-stencil-reduce pattern as a way to simplify the parallel programming of heterogeneous platforms (multicore+GPUs). Loop-of-Stencil-reduce is general enough to subsume map, reduce, map-reduce, stencil, stencil-reduce, and, crucially, their usage in a loop. It transparently targets (by using OpenCL) combinations of CPU cores and GPUs, and it makes it possible to simplify the deployment of a single stencil computation kernel on different GPUs. The paper discusses the implementation of Loop-of-stencil-reduce within the FastFlow parallel framework, considering a simple iterative data-parallel application as running example (Game of Life) and a highly effective parallel filter for visual data restoration to assess performance. Thanks to the high-level design of the Loop-of-stencil-reduce, it was possible to run the filter seamlessly on a multicore machine, on multi-GPUs, and on both.
@inproceedings{opencl:ff:ispa:15,
abstract = {In this paper we advocate the Loop-of-stencil-reduce pattern as a way to simplify the parallel programming of heterogeneous platforms (multicore+GPUs). Loop-of-Stencil-reduce is general enough to subsume map, reduce, map-reduce, stencil, stencil-reduce, and, crucially, their usage in a loop. It transparently targets (by using OpenCL) combinations of CPU cores and GPUs, and it makes it possible to simplify the deployment of a single stencil computation kernel on different GPUs. The paper discusses the implementation of Loop-of-stencil-reduce within the FastFlow parallel framework, considering a simple iterative data-parallel application as running example (Game of Life) and a highly effective parallel filter for visual data restoration to assess performance. Thanks to the high-level design of the Loop-of-stencil-reduce, it was possible to run the filter seamlessly on a multicore machine, on multi-GPUs, and on both.},
address = {Helsinki, Finland},
author = {Aldinucci, Marco and Danelutto, Marco and Drocco, Maurizio and Kilpatrick, Peter and Peretti Pezzi, Guilherme and Torquati, Massimo},
booktitle = {Proc. of Intl. Workshop on Reengineering for Parallelism in Heterogeneous Parallel Platforms (RePara)},
date-added = {2015-07-05 09:48:33 +0000},
date-modified = {2015-09-24 11:14:56 +0000},
doi = {10.1109/Trustcom.2015.628},
keywords = {fastflow, repara, HPC},
month = aug,
pages = {172--177},
publisher = {IEEE},
title = {The Loop-of-Stencil-Reduce paradigm},
url = {https://iris.unito.it/retrieve/handle/2318/1523738/52857/15_RePara_ISPA.pdf},
year = {2015},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1523738/52857/15_RePara_ISPA.pdf},
bdsk-url-2 = {https://doi.org/10.1109/Trustcom.2015.628}} Integrating Multi-omic features exploiting Chromosome Conformation Capture data
Ivan Merelli, Fabio Tordini, Maurizio Drocco, Marco Aldinucci, Pietro Liò, Luciano Milanesi
Frontiers in Genetics
bioinformatics fastflow
The representation, integration and interpretation of omic data is a complex task, in particular considering the huge amount of information that is daily produced in molecular biology laboratories all around the world. The reason is that sequencing data regarding expression profiles, methylation patterns, and chromatin domains is difficult to harmonize in a systems biology view, since genome browsers only allow coordinate-based representations, discarding functional clusters created by the spatial conformation of the DNA in the nucleus. In this context, recent progresses in high throughput molecular biology techniques and bioinformatics have provided insights into chromatin interactions on a larger scale and offer a formidable support for the interpretation of multi-omic data. In particular, a novel sequencing technique called Chromosome Conformation Capture (3C) allows the analysis of the chromosome organization in the cell's natural state. While performed genome wide, this technique is usually called Hi-C. Inspired by service applications such as Google Maps, we developed NuChart, an R package that integrates Hi-C data to describe the chromosomal neighbourhood starting from the information about gene positions, with the possibility of mapping on the achieved graphs genomic features such as methylation patterns and histone modifications, along with expression profiles. In this paper we show the importance of the NuChart application for the integration of multi-omic data in a systems biology fashion, with particular interest in cytogenetic applications of these techniques. Moreover, we demonstrate how the integration of multi-omic data can provide useful information in understanding why genes are in certain specific positions inside the nucleus and how epigenetic patterns correlate with their expression.
@article{nuchart:frontiers:15,
abstract = {The representation, integration and interpretation of omic data is a complex task, in particular considering the huge amount of information that is daily produced in molecular biology laboratories all around the world. The reason is that sequencing data regarding expression profiles, methylation patterns, and chromatin domains is difficult to harmonize in a systems biology view, since genome browsers only allow coordinate-based representations, discarding functional clusters created by the spatial conformation of the DNA in the nucleus. In this context, recent progresses in high throughput molecular biology techniques and bioinformatics have provided insights into chromatin interactions on a larger scale and offer a formidable support for the interpretation of multi-omic data. In particular, a novel sequencing technique called Chromosome Conformation Capture (3C) allows the analysis of the chromosome organization in the cell's natural state. While performed genome wide, this technique is usually called Hi-C. Inspired by service applications such as Google Maps, we developed NuChart, an R package that integrates Hi-C data to describe the chromosomal neighbourhood starting from the information about gene positions, with the possibility of mapping on the achieved graphs genomic features such as methylation patterns and histone modifications, along with expression profiles. In this paper we show the importance of the NuChart application for the integration of multi-omic data in a systems biology fashion, with particular interest in cytogenetic applications of these techniques. Moreover, we demonstrate how the integration of multi-omic data can provide useful information in understanding why genes are in certain specific positions inside the nucleus and how epigenetic patterns correlate with their expression.},
author = {Merelli, Ivan and Tordini, Fabio and Drocco, Maurizio and Aldinucci, Marco and Li{\`o}, Pietro and Milanesi, Luciano},
date-added = {2015-02-01 16:38:47 +0000},
date-modified = {2015-09-24 11:23:10 +0000},
doi = {10.3389/fgene.2015.00040},
issn = {1664-8021},
journal = {Frontiers in Genetics},
keywords = {bioinformatics, fastflow, bioinformatics},
number = {40},
title = {Integrating Multi-omic features exploiting {Chromosome Conformation Capture} data},
url = {http://journal.frontiersin.org/Journal/10.3389/fgene.2015.00040/pdf},
volume = {6},
year = {2015},
bdsk-url-1 = {http://journal.frontiersin.org/Journal/10.3389/fgene.2015.00040/pdf},
bdsk-url-2 = {http://dx.doi.org/10.3389/fgene.2015.00040}} Memory-Optimised Parallel Processing of Hi-C Data
Maurizio Drocco, Claudia Misale, Guilherme Peretti Pezzi, Fabio Tordini, Marco Aldinucci
Proc. of 23rd Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)
bioinformatics fastflow impact paraphrase repara
This paper presents the optimisation efforts on the creation of a graph-based mapping representation of gene adjacency. The method is based on the Hi-C process, starting from Next Generation Sequencing data, and it analyses a huge amount of static data in order to produce maps for one or more genes. Straightforward parallelisation of this scheme does not yield acceptable performance on multicore architectures since the scalability is rather limited due to the memory bound nature of the problem. This work focuses on the memory optimisations that can be applied to the graph construction algorithm and its (complex) data structures to derive a cache-oblivious algorithm and eventually to improve the memory bandwidth utilisation. We used as running example NuChart-II, a tool for annotation and statistic analysis of Hi-C data that creates a gene-centric neighborhood graph. The proposed approach, which is exemplified for Hi-C, addresses several common issue in the parallelisation of memory bound algorithms for multicore. Results show that the proposed approach is able to increase the parallel speedup from 7x to 22x (on a 32-core platform). Finally, the proposed C++ implementation outperforms the first R NuChart prototype, by which it was not possible to complete the graph generation because of strong memory-saturation problems.
@inproceedings{nuchart:speedup:15,
abstract = {This paper presents the optimisation efforts on the creation of a graph-based mapping representation of gene adjacency. The method is based on the Hi-C process, starting from Next Generation Sequencing data, and it analyses a huge amount of static data in order to produce maps for one or more genes. Straightforward parallelisation of this scheme does not yield acceptable performance on multicore architectures since the scalability is rather limited due to the memory bound nature of the problem. This work focuses on the memory optimisations that can be applied to the graph construction algorithm and its (complex) data structures to derive a cache-oblivious algorithm and eventually to improve the memory bandwidth utilisation. We used as running example NuChart-II, a tool for annotation and statistic analysis of Hi-C data that creates a gene-centric neighborhood graph. The proposed approach, which is exemplified for Hi-C, addresses several common issue in the parallelisation of memory bound algorithms for multicore. Results show that the proposed approach is able to increase the parallel speedup from 7x to 22x (on a 32-core platform). Finally, the proposed C++ implementation outperforms the first R NuChart prototype, by which it was not possible to complete the graph generation because of strong memory-saturation problems.},
author = {Drocco, Maurizio and Misale, Claudia and Peretti Pezzi, Guilherme and Tordini, Fabio and Aldinucci, Marco},
booktitle = {Proc. of 23rd Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)},
date-added = {2014-12-03 13:54:08 +0000},
date-modified = {2017-12-12 14:45:09 +0000},
doi = {10.1109/PDP.2015.63},
keywords = {fastflow,bioinformatics, paraphrase, repara, impact},
month = mar,
pages = {1--8},
publisher = {IEEE},
title = {Memory-Optimised Parallel Processing of {Hi-C} Data},
url = {https://iris.unito.it/retrieve/handle/2318/1521910/40615/2015_pdp_memopt.pdf},
year = {2015},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1521910/40615/2015_pdp_memopt.pdf},
bdsk-url-2 = {https://doi.org/10.1109/PDP.2015.63}} Parallel Exploration of the Nuclear Chromosome Conformation with NuChart-II
Fabio Tordini, Maurizio Drocco, Claudia Misale, Luciano Milanesi, Pietro Liò, Ivan Merelli, Marco Aldinucci
Proc. of 23rd Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)
bioinformatics fastflow impact paraphrase repara
High-throughput molecular biology techniques are widely used to identify physical interactions between genetic elements located throughout the human genome. Chromosome Conformation Capture (3C) and other related techniques allow to investigate the spatial organisation of chromosomes in the cell's natural state. Recent results have shown that there is a large correlation between co-localization and co-regulation of genes, but these important information are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we introduce NuChart-II, a tool for Hi-C data analysis that provides a gene-centric view of the chromosomal neighbour- hood in a graph-based manner. NuChart-II is an efficient and highly optimized C++ re-implementation of a previous prototype package developed in R. Representing Hi-C data using a graph-based approach overcomes the common view relying on genomic coordinates and permits the use of graph analysis techniques to explore the spatial conformation of a gene neighbourhood.
@inproceedings{nuchar:tool:15,
abstract = {High-throughput molecular biology techniques are widely used to identify physical interactions between genetic elements located throughout the human genome. Chromosome Conformation Capture (3C) and other related techniques allow to investigate the spatial organisation of chromosomes in the cell's natural state. Recent results have shown that there is a large correlation between co-localization and co-regulation of genes, but these important information are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we introduce NuChart-II, a tool for Hi-C data analysis that provides a gene-centric view of the chromosomal neighbour- hood in a graph-based manner. NuChart-II is an efficient and highly optimized C++ re-implementation of a previous prototype package developed in R. Representing Hi-C data using a graph-based approach overcomes the common view relying on genomic coordinates and permits the use of graph analysis techniques to explore the spatial conformation of a gene neighbourhood.},
author = {Fabio Tordini and Maurizio Drocco and Claudia Misale and Luciano Milanesi and Pietro Li{\`o} and Ivan Merelli and Marco Aldinucci},
booktitle = {Proc. of 23rd Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)},
date-added = {2014-12-03 13:51:17 +0000},
date-modified = {2017-12-12 13:55:10 +0000},
doi = {10.1109/PDP.2015.104},
keywords = {fastflow, bioinformatics, paraphrase, repara, impact},
month = mar,
publisher = {IEEE},
title = {Parallel Exploration of the Nuclear Chromosome Conformation with {NuChart-II}},
url = {https://iris.unito.it/retrieve/handle/2318/1522038/40619/2015_pdp_nuchartff.pdf},
year = {2015},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1522038/40619/2015_pdp_nuchartff.pdf},
bdsk-url-2 = {https://doi.org/10.1109/PDP.2015.104}} High-Performance Haplotype Assembly
Marco Aldinucci, Andrea Bracciali, Tobias Marschall, Murray Patterson, Nadia Pisanti, Massimo Torquati
Computational Intelligence Methods for Bioinformatics and Biostatistics - 11th International Meeting, CIBB 2014, Cambridge, UK, June 26-28, 2014, Revised Selected Papers
bioinformatics fastflow
The problem of Haplotype Assembly is an essential step in human genome analysis. It is typically formalised as the Minimum Error Correction (MEC) problem which is NP-hard. MEC has been approached using heuristics, integer linear programming, and fixed-parameter tractability (FPT), including approaches whose runtime is exponential in the length of the DNA fragments obtained by the sequencing process. Technological improvements are currently increasing fragment length, which drastically elevates computational costs for such methods. We present pWhatsHap, a multi-core parallelisation of WhatsHap, a recent FPT optimal approach to MEC. WhatsHap moves complexity from fragment length to fragment overlap and is hence of particular interest when considering sequencing technology's current trends. pWhatsHap further improves the efficiency in solving the MEC problem, as shown by experiments performed on datasets with high coverage.
@inproceedings{14:ff:whatsapp:cibb,
abstract = {The problem of Haplotype Assembly is an essential step in human genome analysis. It is typically formalised as the Minimum Error Correction (MEC) problem which is NP-hard. MEC has been approached using heuristics, integer linear programming, and fixed-parameter tractability (FPT), including approaches whose runtime is exponential in the length of the DNA fragments obtained by the sequencing process. Technological improvements are currently increasing fragment length, which drastically elevates computational costs for such methods. We present pWhatsHap, a multi-core parallelisation of WhatsHap, a recent FPT optimal approach to MEC. WhatsHap moves complexity from fragment length to fragment overlap and is hence of particular interest when considering sequencing technology's current trends. pWhatsHap further improves the efficiency in solving the MEC problem, as shown by experiments performed on datasets with high coverage.},
address = {Cambridge, UK},
author = {Marco Aldinucci and Andrea Bracciali and Tobias Marschall and Murray Patterson and Nadia Pisanti and Massimo Torquati},
booktitle = {Computational Intelligence Methods for Bioinformatics and Biostatistics - 11th International Meeting, {CIBB} 2014, Cambridge, UK, June 26-28, 2014, Revised Selected Papers},
date-added = {2014-12-01 23:07:21 +0000},
date-modified = {2016-08-20 14:15:59 +0000},
doi = {10.1007/978-3-319-24462-4_21},
editor = {Clelia Di Serio and Pietro Li{\`{o}} and Alessandro Nonis and Roberto Tagliaferri},
keywords = {fastflow, bioinformatics},
pages = {245--258},
publisher = {Springer},
series = {{LNCS}},
title = {High-Performance Haplotype Assembly},
url = {https://iris.unito.it/retrieve/handle/2318/1523292/46714/2014_pHaplo_cibb.pdf},
volume = {8623},
year = {2015},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1523292/46714/2014_pHaplo_cibb.pdf},
bdsk-url-2 = {https://doi.org/10.1007/978-3-319-24462-4_21}} Parallel Visual Data Restoration on Multi-GPGPUs using Stencil-Reduce Pattern
Marco Aldinucci, Guilherme Peretti Pezzi, Maurizio Drocco, Concetto Spampinato, Massimo Torquati
International Journal of High Performance Computing Applications
fastflow HPC impact paraphrase
In this paper, a highly effective parallel filter for visual data restoration is presented. The filter is designed following a skeletal approach, using a newly proposed stencil-reduce, and has been implemented by way of the FastFlow parallel programming library. As a result of its high-level design, it is possible to run the filter seamlessly on a multicore machine, on multi-GPGPUs, or on both. The design and implementation of the filter are discussed, and an experimental evaluation is presented.
@article{ff:denoiser:ijhpca:15,
abstract = {In this paper, a highly effective parallel filter for visual data restoration is presented. The filter is designed following a skeletal approach, using a newly proposed stencil-reduce, and has been implemented by way of the FastFlow parallel programming library. As a result of its high-level design, it is possible to run the filter seamlessly on a multicore machine, on multi-GPGPUs, or on both. The design and implementation of the filter are discussed, and an experimental evaluation is presented.},
author = {Aldinucci, Marco and Peretti Pezzi, Guilherme and Drocco, Maurizio and Spampinato, Concetto and Torquati, Massimo},
date-added = {2014-08-23 00:06:10 +0000},
date-modified = {2015-09-24 11:21:20 +0000},
doi = {10.1177/1094342014567907},
journal = {International Journal of High Performance Computing Applications},
keywords = {fastflow, paraphrase, impact, HPC},
number = {4},
pages = {461--472},
title = {Parallel Visual Data Restoration on Multi-{GPGPUs} using Stencil-Reduce Pattern},
url = {https://iris.unito.it/retrieve/handle/2318/1522073/299200/ijhpca_4aperto.pdf},
volume = {29},
year = {2015},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/1522073/299200/ijhpca_4aperto.pdf},
bdsk-url-2 = {https://doi.org/10.1177/1094342014567907}} NuChart-II: a graph-based approach for the analysis and interpretation of Hi-C data
Fabio Tordini, Maurizio Drocco, Ivan Merelli, Luciano Milanesi, Pietro Liò, Marco Aldinucci
Proc. of 11th Intl. Meeting on Computational Intelligence Methods for Bioinformatics and Biostatistics (CIBB)
bioinformatics fastflow paraphrase repara
Long-range chromosomal associations between genomic regions, and their repositioning in the 3D space of the nucleus, are now considered to be key contributors to the regulation of gene expressions, and important links have been highlighted with other genomic features involved in DNA rearrangements. Recent Chromosome Conformation Capture (3C) measurements performed with high throughput sequencing (Hi-C) and molecular dynamics studies show that there is a large correlation between co-localization and co-regulation of genes, but these important researches are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we present NuChart-II, a software that allows the user to annotate and visualize a list of input genes with information relying on Hi-C data, integrating knowledge data about genomic features that are involved in the chromosome spatial organization. This software works directly with sequenced reads to identify related Hi-C fragments, with the aim of creating gene-centric neighbourhood graphs on which multi-omics features can be mapped. NuChart-II is a highly optimized implementation of a previous prototype package developed in R, in which the graph-based representation of Hi-C data was tested. The prototype showed inevitable problems of scalability while working genome-wide on large datasets: particular attention has been paid in optimizing the data structures employed while constructing the neighbourhood graph, so as to foster an efficient parallel implementation of the software. The normalization of Hi-C data has been modified and improved, in order to provide a reliable estimation of proximity likelihood for the genes.
@inproceedings{14:ff:nuchart:cibb,
abstract = {Long-range chromosomal associations between genomic regions, and their repositioning in the 3D space of the nucleus, are now considered to be key contributors to the regulation of gene expressions, and important links have been highlighted with other genomic features involved in DNA rearrangements. Recent Chromosome Conformation Capture (3C) measurements performed with high throughput sequencing (Hi-C) and molecular dynamics studies show that there is a large correlation between co-localization and co-regulation of genes, but these important researches are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we present NuChart-II, a software that allows the user to annotate and visualize a list of input genes with information relying on Hi-C data, integrating knowledge data about genomic features that are involved in the chromosome spatial organization. This software works directly with sequenced reads to identify related Hi-C fragments, with the aim of creating gene-centric neighbourhood graphs on which multi-omics features can be mapped. NuChart-II is a highly optimized implementation of a previous prototype package developed in R, in which the graph-based representation of Hi-C data was tested. The prototype showed inevitable problems of scalability while working genome-wide on large datasets: particular attention has been paid in optimizing the data structures employed while constructing the neighbourhood graph, so as to foster an efficient parallel implementation of the software. The normalization of Hi-C data has been modified and improved, in order to provide a reliable estimation of proximity likelihood for the genes.},
address = {Cambridge, UK},
author = {Fabio Tordini and Maurizio Drocco and Ivan Merelli and Luciano Milanesi and Pietro Li{\`o} and Marco Aldinucci},
booktitle = {Proc. of 11th Intl. Meeting on Computational Intelligence Methods for Bioinformatics and Biostatistics (CIBB)},
date-modified = {2017-12-12 15:19:25 +0000},
doi = {10.1007/978-3-319-24462-4_25},
editor = {Clelia Di Serio and Pietro Li{\`{o}} and Alessandro Nonis and Roberto Tagliaferri},
isbn = {978-3-319-24461-7},
keywords = {fastflow, bioinformatics, paraphrase, repara, bioinformatics},
month = jun,
pages = {298--311},
publisher = {Springer},
series = {{LNCS}},
title = {{NuChart-II}: a graph-based approach for the analysis and interpretation of {Hi-C} data},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_nuchart_cibb.pdf},
volume = {8623},
year = {2015},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2014_nuchart_cibb.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-319-24462-4_25}} 2014
2014
Parallel stochastic systems biology in the cloud
Marco Aldinucci, Massimo Torquati, Concetto Spampinato, Maurizio Drocco, Claudia Misale, Cristina Calcagno, Mario Coppo
Briefings in Bioinformatics
bioinformatics fastflow impact paraphrase
The stochastic modelling of biological systems, coupled with Monte Carlo simulation of models, is an increasingly popular technique in bioinformatics. The simulation-analysis workflow may result computationally expensive reducing the interactivity required in the model tuning. In this work, we advocate the high-level software design as a vehicle for building efficient and portable parallel simulators for the cloud. In particular, the Calculus of Wrapped Components (CWC) simulator for systems biology, which is designed according to the FastFlow pattern-based approach, is presented and discussed. Thanks to the FastFlow framework, the CWC simulator is designed as a high-level workflow that can simulate CWC models, merge simulation results and statistically analyse them in a single parallel workflow in the cloud. To improve interactivity, successive phases are pipelined in such a way that the workflow begins to output a stream of analysis results immediately after simulation is started. Performance and effectiveness of the CWC simulator are validated on the Amazon Elastic Compute Cloud.
@article{cwc:cloud:bib:13,
abstract = {The stochastic modelling of biological systems, coupled with Monte Carlo simulation of models, is an increasingly popular technique in bioinformatics. The simulation-analysis workflow may result computationally expensive reducing the interactivity required in the model tuning. In this work, we advocate the high-level software design as a vehicle for building efficient and portable parallel simulators for the cloud. In particular, the Calculus of Wrapped Components (CWC) simulator for systems biology, which is designed according to the FastFlow pattern-based approach, is presented and discussed. Thanks to the FastFlow framework, the CWC simulator is designed as a high-level workflow that can simulate CWC models, merge simulation results and statistically analyse them in a single parallel workflow in the cloud. To improve interactivity, successive phases are pipelined in such a way that the workflow begins to output a stream of analysis results immediately after simulation is started. Performance and effectiveness of the CWC simulator are validated on the Amazon Elastic Compute Cloud.},
author = {Marco Aldinucci and Massimo Torquati and Concetto Spampinato and Maurizio Drocco and Claudia Misale and Cristina Calcagno and Mario Coppo},
date-added = {2014-12-21 17:49:54 +0000},
date-modified = {2021-04-24 23:24:25 +0200},
doi = {10.1093/bib/bbt040},
issn = {1467-5463},
journal = {Briefings in Bioinformatics},
keywords = {fastflow, bioinformatics, paraphrase, impact},
number = {5},
pages = {798--813},
title = {Parallel stochastic systems biology in the cloud},
url = {https://iris.unito.it/retrieve/handle/2318/140080/22528/FF_Cloud_briefings_final_submitted_copy.pdf},
volume = {15},
year = {2014},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/140080/22528/FF_Cloud_briefings_final_submitted_copy.pdf},
bdsk-url-2 = {https://doi.org/10.1093/bib/bbt040}} Design patterns percolating to parallel programming framework implementation
Marco Aldinucci, Sonia Campa, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
International Journal of Parallel Programming
fastflow paraphrase
Structured parallel programming is recognised as a viable and effective means of tackling parallel programming problems. Recently, a set of simple and powerful parallel building blocks (RISC-pb2l) has been proposed to support modelling and implementation of parallel frameworks. In this work we demonstrate how that same parallel building block set may be used to model both general purpose parallel programming abstractions, not usually listed in classical skeleton sets, and more specialized domain specific parallel patterns. We show how an implementation of RISC-pb2l can be realised via the FastFlow framework and present experimental evidence of the feasibility and efficiency of the approach.
@article{ijpp:patterns:13,
abstract = {Structured parallel programming is recognised as a viable and effective means of tackling parallel programming problems. Recently, a set of simple and powerful parallel building blocks (RISC-pb2l) has been proposed to support modelling and implementation of parallel frameworks. In this work we demonstrate how that same parallel building block set may be used to model both general purpose parallel programming abstractions, not usually listed in classical skeleton sets, and more specialized domain specific parallel patterns. We show how an implementation of RISC-pb2l can be realised via the FastFlow framework and present experimental evidence of the feasibility and efficiency of the approach.},
author = {Marco Aldinucci and Sonia Campa and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
date-added = {2014-12-21 17:47:21 +0000},
date-modified = {2015-09-27 12:32:37 +0000},
doi = {10.1007/s10766-013-0273-6},
issn = {0885-7458},
journal = {International Journal of Parallel Programming},
keywords = {fastflow, paraphrase},
number = {6},
pages = {1012--1031},
title = {Design patterns percolating to parallel programming framework implementation},
url = {https://iris.unito.it/retrieve/handle/2318/140069/22527/2013_ijpp_patterns-web_4aperto_1238811.pdf},
volume = {42},
year = {2014},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/140069/22527/2013_ijpp_patterns-web_4aperto_1238811.pdf},
bdsk-url-2 = {https://doi.org/10.1007/s10766-013-0273-6}} Decision Tree Building on Multi-Core using FastFlow
Marco Aldinucci, Salvatore Ruggieri, Massimo Torquati
Concurrency and Computation: Practice and Experience
fastflow paraphrase
The whole computer hardware industry embraced multi-core. The extreme optimisation of sequential algorithms is then no longer sufficient to squeeze the real machine power, which can be only exploited via thread-level parallelism. Decision tree algorithms exhibit natural concurrency that makes them suitable to be parallelised. This paper presents an in-depth study of the parallelisation of an implementation of the C4.5 algorithm for multi-core architectures. We characterise elapsed time lower bounds for the forms of parallelisations adopted, and achieve close to optimal performances. Our implementation is based on the FastFlow parallel programming environment and it requires minimal changes to the original sequential code.
@article{yadtff:ccpe:13,
abstract = {The whole computer hardware industry embraced multi-core. The extreme optimisation of sequential algorithms is then no longer sufficient to squeeze the real machine power, which can be only exploited via thread-level parallelism. Decision tree algorithms exhibit natural concurrency that makes them suitable to be parallelised. This paper presents an in-depth study of the parallelisation of an implementation of the C4.5 algorithm for multi-core architectures. We characterise elapsed time lower bounds for the forms of parallelisations adopted, and achieve close to optimal performances. Our implementation is based on the FastFlow parallel programming environment and it requires minimal changes to the original sequential code.},
author = {Marco Aldinucci and Salvatore Ruggieri and Massimo Torquati},
date-added = {2014-12-21 17:46:33 +0000},
date-modified = {2015-09-27 12:17:52 +0000},
doi = {10.1002/cpe.3063},
journal = {Concurrency and Computation: Practice and Experience},
keywords = {fastflow, paraphrase},
number = {3},
pages = {800--820},
title = {Decision Tree Building on Multi-Core using FastFlow},
url = {https://iris.unito.it/retrieve/handle/2318/139522/118602/yadtff-j.pdf},
volume = {26},
year = {2014},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/139522/118602/yadtff-j.pdf},
bdsk-url-2 = {https://doi.org/10.1002/cpe.3063}} On designing multicore-aware simulators for systems biology endowed with on-line statistics
Marco Aldinucci, Cristina Calcagno, Mario Coppo, Ferruccio Damiani, Maurizio Drocco, Eva Sciacca, Salvatore Spinella, Massimo Torquati, Angelo Troina
BioMed Research International
bioinformatics fastflow paraphrase
The paper arguments are on enabling methodologies for the design of a fully parallel, online, interactive tool aiming to support the bioinformatics scientists .In particular, the features of these methodologies, supported by the FastFlow parallel programming framework, are shown on a simulation tool to perform the modeling, the tuning, and the sensitivity analysis of stochastic biological models. A stochastic simulation needs thousands of independent simulation trajectories turning into big data that should be analysed by statistic and data mining tools. In the considered approach the two stages are pipelined in such a way that the simulation stage streams out the partial results of all simulation trajectories to the analysis stage that immediately produces a partial result. The simulation-analysis workflow is validated for performance and effectiveness of the online analysis in capturing biological systems behavior on a multicore platform and representative proof-of-concept biological systems. The exploited methodologies include pattern-based parallel programming and data streaming that provide key features to the software designers such as performance portability and efficient in-memory (big) data management and movement. Two paradigmatic classes of biological systems exhibiting multistable and oscillatory behavior are used as a testbed.
@article{cwcsim:ff:multicore:biomed:14,
abstract = {The paper arguments are on enabling methodologies for the design of a fully parallel, online, interactive tool aiming to support the bioinformatics scientists .In particular, the features of these methodologies, supported by the FastFlow parallel programming framework, are shown on a simulation tool to perform the modeling, the tuning, and the sensitivity analysis of stochastic biological models. A stochastic simulation needs thousands of independent simulation trajectories turning into big data that should be analysed by statistic and data mining tools. In the considered approach the two stages are pipelined in such a way that the simulation stage streams out the partial results of all simulation trajectories to the analysis stage that immediately produces a partial result. The simulation-analysis workflow is validated for performance and effectiveness of the online analysis in capturing biological systems behavior on a multicore platform and representative proof-of-concept biological systems. The exploited methodologies include pattern-based parallel programming and data streaming that provide key features to the software designers such as performance portability and efficient in-memory (big) data management and movement. Two paradigmatic classes of biological systems exhibiting multistable and oscillatory behavior are used as a testbed.},
author = {Marco Aldinucci and Cristina Calcagno and Mario Coppo and Ferruccio Damiani and Maurizio Drocco and Eva Sciacca and Salvatore Spinella and Massimo Torquati and Angelo Troina},
date-added = {2014-06-26 21:30:32 +0000},
date-modified = {2015-09-27 12:17:05 +0000},
doi = {10.1155/2014/207041},
journal = {BioMed Research International},
keywords = {fastflow, bioinformatics, paraphrase},
title = {On designing multicore-aware simulators for systems biology endowed with on-line statistics},
url = {http://downloads.hindawi.com/journals/bmri/2014/207041.pdf},
year = {2014},
bdsk-url-1 = {http://downloads.hindawi.com/journals/bmri/2014/207041.pdf},
bdsk-url-2 = {https://doi.org/10.1155/2014/207041}} FastFlow: Combining Pattern-Level Abstraction and Efficiency in GPGPUs
Marco Aldinucci, Massimo Torquati, Maurizio Drocco, Guilherme Peretti Pezzi, Concetto Spampinato
GPU Technology Conference (GTC)
fastflow HPC impact paraphrase
Learn how FastFlow's parallel patterns can be used to design parallel applications for execution on both CPUs and GPGPUs while avoiding most of the complex low-level detail needed to make them efficient, portable and rapid to prototype. As use case, we will show the design and effectiveness of a novel universal image filtering template based on the variational approach.
@inproceedings{ff:gtc:2014,
abstract = {Learn how FastFlow's parallel patterns can be used to design parallel applications for execution on both CPUs and GPGPUs while avoiding most of the complex low-level detail needed to make them efficient, portable and rapid to prototype. As use case, we will show the design and effectiveness of a novel universal image filtering template based on the variational approach.},
address = {San Jose, CA, USA},
author = {Aldinucci, Marco and Torquati, Massimo and Drocco, Maurizio and Peretti Pezzi, Guilherme and Spampinato, Concetto},
booktitle = {GPU Technology Conference (GTC)},
date-added = {2014-04-19 12:52:40 +0000},
date-modified = {2017-12-12 13:54:25 +0000},
keywords = {fastflow, HPC, impact, paraphrase},
month = mar,
title = {FastFlow: Combining Pattern-Level Abstraction and Efficiency in {GPGPUs}},
url = {http://calvados.di.unipi.it/storage/talks/2014_S4729-Marco-Aldinucci.pdf},
year = {2014},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/talks/2014_S4729-Marco-Aldinucci.pdf}} Exercising high-level parallel programming on streams: a systems biology use case
Marco Aldinucci, Maurizio Drocco, Guilherme Peretti Pezzi, Claudia Misale, Fabio Tordini, Massimo Torquati
Proc. of 34th IEEE Intl. Conference on Distributed Computing Systems Workshops (ICDCSW)
bioinformatics fastflow impact paraphrase
The stochastic modelling of biological systems, cou- pled with Monte Carlo simulation of models, is an increasingly popular technique in Bioinformatics. The simulation-analysis workflow may result into a computationally expensive task reducing the interactivity required in the model tuning. In this work, we advocate high-level software design as a vehicle for building efficient and portable parallel simulators for a variety of platforms, ranging from multi-core platforms to GPGPUs to cloud. In particular, the Calculus of Wrapped Compartments (CWC) parallel simulator for systems biology equipped with on- line mining of results, which is designed according to the FastFlow pattern-based approach, is discussed as a running example. In this work, the CWC simulator is used as a paradigmatic example of a complex C++ application where the quality of results is correlated with both computation and I/O bounds, and where high-quality results might turn into big data. The FastFlow parallel programming framework, which advocates C++ pattern- based parallel programming makes it possible to develop portable parallel code without relinquish neither run-time efficiency nor performance tuning opportunities. Performance and effectiveness of the approach are validated on a variety of platforms, inter-alia cache-coherent multi-cores, cluster of multi-core (Ethernet and Infiniband) and the Amazon Elastic Compute Cloud.
@inproceedings{cwc:gpu:dcperf:14,
abstract = {The stochastic modelling of biological systems, cou- pled with Monte Carlo simulation of models, is an increasingly popular technique in Bioinformatics. The simulation-analysis workflow may result into a computationally expensive task reducing the interactivity required in the model tuning. In this work, we advocate high-level software design as a vehicle for building efficient and portable parallel simulators for a variety of platforms, ranging from multi-core platforms to GPGPUs to cloud. In particular, the Calculus of Wrapped Compartments (CWC) parallel simulator for systems biology equipped with on- line mining of results, which is designed according to the FastFlow pattern-based approach, is discussed as a running example. In this work, the CWC simulator is used as a paradigmatic example of a complex C++ application where the quality of results is correlated with both computation and I/O bounds, and where high-quality results might turn into big data. The FastFlow parallel programming framework, which advocates C++ pattern- based parallel programming makes it possible to develop portable parallel code without relinquish neither run-time efficiency nor performance tuning opportunities. Performance and effectiveness of the approach are validated on a variety of platforms, inter-alia cache-coherent multi-cores, cluster of multi-core (Ethernet and Infiniband) and the Amazon Elastic Compute Cloud.},
address = {Madrid, Spain},
author = {Aldinucci, Marco and Drocco, Maurizio and Peretti Pezzi, Guilherme and Misale, Claudia and Tordini, Fabio and Torquati, Massimo},
booktitle = {Proc. of 34th IEEE Intl. Conference on Distributed Computing Systems Workshops (ICDCSW)},
date-added = {2014-04-19 12:44:39 +0000},
date-modified = {2021-04-24 23:23:10 +0200},
doi = {10.1109/ICDCSW.2014.38},
keywords = {fastflow, bioinformatics, paraphrase, impact},
publisher = {IEEE},
title = {Exercising high-level parallel programming on streams: a systems biology use case},
url = {https://iris.unito.it/retrieve/handle/2318/154516/26657/2014_dcperf_cwc_gpu.pdf},
year = {2014},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/154516/26657/2014_dcperf_cwc_gpu.pdf},
bdsk-url-2 = {https://doi.org/10.1109/ICDCSW.2014.38}} Parallel video denoising on heterogeneous platforms
Marco Aldinucci, Guilherme Peretti Pezzi, Maurizio Drocco, Fabio Tordini, Peter Kilpatrick, Massimo Torquati
Proc. of Intl. Workshop on High-level Programming for Heterogeneous and Hierarchical Parallel Systems (HLPGPU)
fastflow impact paraphrase
In this paper, a highly-effective parallel filter for video denoising is presented. The filter is designed using a skeletal approach, and has been implemented by way of the FastFlow parallel programming library. As a result of its high-level design, it is possible to run the filter seamlessly on a multi-core machine, on GPGPU(s), or on both. The design and the implementation of the filter are discussed, and an experimental evaluation is presented. Various mappings of the filtering stages are comparatively discussed.
@inproceedings{ff:video:hlpgpu:14,
abstract = {In this paper, a highly-effective parallel filter for video denoising is presented. The filter is designed using a skeletal approach, and has been implemented by way of the FastFlow parallel programming library. As a result of its high-level design, it is possible to run the filter seamlessly on a multi-core machine, on GPGPU(s), or on both. The design and the implementation of the filter are discussed, and an experimental evaluation is presented. Various mappings of the filtering stages are comparatively discussed.},
author = {Aldinucci, Marco and Peretti Pezzi, Guilherme and Drocco, Maurizio and Tordini, Fabio and Kilpatrick, Peter and Torquati, Massimo},
booktitle = {Proc. of Intl. Workshop on High-level Programming for Heterogeneous and Hierarchical Parallel Systems (HLPGPU)},
date-added = {2013-12-07 18:28:32 +0000},
date-modified = {2015-09-27 12:42:02 +0000},
keywords = {fastflow, paraphrase, impact},
title = {Parallel video denoising on heterogeneous platforms},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_ff_video_denoiser_hlpgpu.pdf},
year = {2014},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2014_ff_video_denoiser_hlpgpu.pdf}} Accelerating Bowtie2 with a lock-less concurrency approach and memory affinity
Claudia Misale
Proc. of Intl. Euromicro PDP 2014: Parallel Distributed and network-based Processing
fastflow paraphrase
The implementation of DNA alignment tools for Bioinformatics lead to face different problems that dip into performances. A single alignment takes an amount of time that is not predictable and there are different factors that can affect performances, for instance the length of sequences can determine the computational grain of the task and mismatches or insertion/deletion (indels) increase time needed to complete an alignment. Moreover, an alignment is a strong memory- bound problem because of the irregular memory access pat- terns and limitations in memory-bandwidth. Over the years, many alignment tools were implemented. A concrete example is Bowtie2, one of the fastest (concurrent, Pthread-based) and state of the art not GPU-based alignment tool. Bowtie2 exploits concurrency by instantiating a pool of threads, which have access to a global input dataset, share the reference genome and have access to different objects for collecting alignment results. In this paper a modified implementation of Bowtie2 is presented, in which the concurrency structure has been changed. The proposed implementation exploits the task-farm skeleton pattern implemented as a Master-Worker. The Master-Worker pattern permits to delegate only to the Master thread dataset reading and to make private to each Worker data structures that are shared in the original version. Only the reference genome is left shared. As a further optimisation, the Master and each Worker were pinned on cores and the reference genome was allocated interleaved among memory nodes. The proposed implementation is able to gain up to 10 speedup points over the original implementation.
@inproceedings{ff:bowtie2:pdp:14,
abstract = {The implementation of DNA alignment tools for Bioinformatics lead to face different problems that dip into performances. A single alignment takes an amount of time that is not predictable and there are different factors that can affect performances, for instance the length of sequences can determine the computational grain of the task and mismatches or insertion/deletion (indels) increase time needed to complete an alignment. Moreover, an alignment is a strong memory- bound problem because of the irregular memory access pat- terns and limitations in memory-bandwidth. Over the years, many alignment tools were implemented. A concrete example is Bowtie2, one of the fastest (concurrent, Pthread-based) and state of the art not GPU-based alignment tool. Bowtie2 exploits concurrency by instantiating a pool of threads, which have access to a global input dataset, share the reference genome and have access to different objects for collecting alignment results. In this paper a modified implementation of Bowtie2 is presented, in which the concurrency structure has been changed. The proposed implementation exploits the task-farm skeleton pattern implemented as a Master-Worker. The Master-Worker pattern permits to delegate only to the Master thread dataset reading and to make private to each Worker data structures that are shared in the original version. Only the reference genome is left shared. As a further optimisation, the Master and each Worker were pinned on cores and the reference genome was allocated interleaved among memory nodes. The proposed implementation is able to gain up to 10 speedup points over the original implementation.},
address = {Torino, Italy},
author = {Claudia Misale},
booktitle = {Proc. of Intl. Euromicro PDP 2014: Parallel Distributed and network-based Processing},
date-added = {2013-12-07 18:25:55 +0000},
date-modified = {2015-09-27 12:41:24 +0000},
doi = {10.1109/PDP.2014.50},
editor = {Marco Aldinucci and Daniele D'Agostino and Peter Kilpatrick},
keywords = {fastflow, paraphrase},
note = {(Best paper award)},
publisher = {IEEE},
title = {Accelerating Bowtie2 with a lock-less concurrency approach and memory affinity},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_pdp_bowtieff.pdf},
year = {2014},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2014_pdp_bowtieff.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1109/PDP.2014.50}} Message passing on InfiniBand RDMA for parallel run-time supports
Alessandro Secco, Irfan Uddin, Guilherme Peretti Pezzi, Massimo Torquati
Proc. of Intl. Euromicro PDP 2014: Parallel Distributed and network-based Processing
fastflow impact paraphrase
InfiniBand networks are commonly used in the high performance computing area. They offer RDMA-based opera- tions that help to improve the performance of communication subsystems. In this paper, we propose a minimal message-passing communication layer providing the programmer with a point-to- point communication channel implemented by way of InfiniBand RDMA features. Differently from other libraries exploiting the InfiniBand features, such as the well-known Message Passing Interface (MPI), the proposed library is a communication layer only rather than a programming model, and can be easily used as building block for high-level parallel programming frameworks. Evaluated on micro-benchmarks, the proposed RDMA-based communication channel implementation achieves a comparable performance with highly optimised MPI/InfiniBand implemen- tations. Eventually, the flexibility of the communication layer is evaluated by integrating it within the FastFlow parallel frame- work, currently supporting TCP/IP networks (via the ZeroMQ communication library).
@inproceedings{ff:infiniband:pdp:14,
abstract = {InfiniBand networks are commonly used in the high performance computing area. They offer RDMA-based opera- tions that help to improve the performance of communication subsystems. In this paper, we propose a minimal message-passing communication layer providing the programmer with a point-to- point communication channel implemented by way of InfiniBand RDMA features. Differently from other libraries exploiting the InfiniBand features, such as the well-known Message Passing Interface (MPI), the proposed library is a communication layer only rather than a programming model, and can be easily used as building block for high-level parallel programming frameworks. Evaluated on micro-benchmarks, the proposed RDMA-based communication channel implementation achieves a comparable performance with highly optimised MPI/InfiniBand implemen- tations. Eventually, the flexibility of the communication layer is evaluated by integrating it within the FastFlow parallel frame- work, currently supporting TCP/IP networks (via the ZeroMQ communication library).},
address = {Torino, Italy},
author = {Secco, Alessandro and Uddin, Irfan and Peretti Pezzi, Guilherme and Torquati, Massimo},
booktitle = {Proc. of Intl. Euromicro PDP 2014: Parallel Distributed and network-based Processing},
date-added = {2013-12-07 18:22:35 +0000},
date-modified = {2015-09-27 12:35:04 +0000},
doi = {10.1109/PDP.2014.23},
editor = {Marco Aldinucci and Daniele D'Agostino and Peter Kilpatrick},
keywords = {fastflow, paraphrase, impact},
publisher = {IEEE},
title = {Message passing on InfiniBand {RDMA} for parallel run-time supports},
url = {https://iris.unito.it/retrieve/handle/2318/151178/690885/2014_ff_infiniband_pdp.pdf},
year = {2014},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2014_ff_infiniband_pdp.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1109/PDP.2014.23}} A Lightweight Run-Time Support For Fast Dense Linear Algebra on Multi-Core
Daniele Buono, Marco Danelutto, Tiziano De Matteis, Gabriele Mencagli, Massimo Torquati
Proc. of the 12th International Conference on Parallel and Distributed Computing and Networks (PDCN 2014)
fastflow
@inproceedings{ff:ffmdf:pdcn:14,
author = {Buono, Daniele and Danelutto, Marco and De Matteis, Tiziano and Mencagli, Gabriele and Torquati, Massimo},
booktitle = {Proc. of the 12th International Conference on Parallel and Distributed Computing and Networks (PDCN 2014)},
date-modified = {2018-12-27 18:52:39 +0100},
keywords = {fastflow},
month = feb,
publisher = {IASTED, ACTA press},
title = {A Lightweight Run-Time Support For Fast Dense Linear Algebra on Multi-Core},
year = {2014}} A Dynamic Memory Allocator for heterogeneous platforms
Maurizio Drocco, Marco Aldinucci, Massimo Torquati
Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) – Poster Abstracts
fastflow HPC
Modern computers are built upon heterogeneous multi-core/many cores architectures (e.g. GPGPU connected to multi-core CPU). Achieving peak performance on these architectures is hard and may require a substantial programming effort. High-level programming patterns, coupled with efficient low-level runtime supports, have been proposed to relieve the programmer from worrying about low-level details such as synchronisation of racing processes as well as those fine tunings needed to improve the overall performance. Among them are (parallel) dynamic memory allocation and effective exploitation of the memory hierarchy. The memory allocator is often a bottleneck that severely limits program scalability, robustness and portability on parallel systems. In this work we introduce a novel memory allocator, based on the FastFlow's allocator and the recently proposed CUDA Unified Memory, which aims to efficiently integrate host and device memories into a unique dynamic-allocable memory space, accessible transparently by both host and device code.
@inproceedings{ff:acaces:14,
abstract = {Modern computers are built upon heterogeneous multi-core/many cores architectures (e.g. GPGPU connected to multi-core CPU). Achieving peak performance on these architectures is hard and may require a substantial programming effort. High-level programming patterns, coupled with efficient low-level runtime supports, have been proposed to relieve the programmer from worrying about low-level details such as synchronisation of racing processes as well as those fine tunings needed to improve the overall performance. Among them are (parallel) dynamic memory allocation and effective exploitation of the memory hierarchy. The memory allocator is often a bottleneck that severely limits program scalability, robustness and portability on parallel systems. In this work we introduce a novel memory allocator, based on the FastFlow's allocator and the recently proposed CUDA Unified Memory, which aims to efficiently integrate host and device memories into a unique dynamic-allocable memory space, accessible transparently by both host and device code.},
address = {Fiuggi, Italy},
author = {Maurizio Drocco and Marco Aldinucci and Massimo Torquati},
booktitle = {Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) -- Poster Abstracts},
date-modified = {2016-08-20 17:29:47 +0000},
keywords = {fastflow, HPC},
publisher = {HiPEAC},
title = {A Dynamic Memory Allocator for heterogeneous platforms},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_ACACES_ex-abstract.pdf},
year = {2014},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2013_ACACES_ex-abstract.pdf},
bdsk-url-2 = {http://calvados.di.unipi.it/storage/paper_files/2014_ACACES_ex-abstract.pdf}} Sequence alignment tools: one parallel pattern to rule them all?
Claudia Misale, Giulio Ferrero, Massimo Torquati, Marco Aldinucci
BioMed Research International
bioinformatics fastflow paraphrase repara
In this paper we advocate high-level programming methodology for Next Generation Sequencers (NGS) alignment tools for both productivity and absolute performance. We analyse the problem of parallel alignment and review the parallelisation strategies of the most popular alignment tools, which can all be abstracted to a single parallel paradigm. We compare these tools against their porting onto the FastFlow pattern-based programming framework, which provides programmers with high-level parallel patterns. By using a high-level approach, programmers are liberated from all complex aspects of parallel programming, such as synchronisation protocols and task scheduling, gaining more possibility for seamless performance tuning. In this work we show some use case in which, by using a high-level approach for parallelising NGS tools, it is possible to obtain comparable or even better absolute performance for all used datasets.
@article{bowtie-bwa:ff:multicore:biomed:14,
abstract = {In this paper we advocate high-level programming methodology for Next Generation Sequencers (NGS) alignment tools for both productivity and absolute performance. We analyse the problem of parallel alignment and review the parallelisation strategies of the most popular alignment tools, which can all be abstracted to a single parallel paradigm. We compare these tools against their porting onto the FastFlow pattern-based programming framework, which provides programmers with high-level parallel patterns. By using a high-level approach, programmers are liberated from all complex aspects of parallel programming, such as synchronisation protocols and task scheduling, gaining more possibility for seamless performance tuning. In this work we show some use case in which, by using a high-level approach for parallelising NGS tools, it is possible to obtain comparable or even better absolute performance for all used datasets.},
author = {Claudia Misale and Giulio Ferrero and Massimo Torquati and Marco Aldinucci},
date-added = {2013-01-15 15:55:59 +0000},
date-modified = {2015-09-27 12:16:28 +0000},
doi = {10.1155/2014/539410},
journal = {BioMed Research International},
keywords = {fastflow,bioinformatics, paraphrase, repara},
title = {Sequence alignment tools: one parallel pattern to rule them all?},
url = {http://downloads.hindawi.com/journals/bmri/2014/539410.pdf},
year = {2014},
bdsk-url-1 = {http://downloads.hindawi.com/journals/bmri/2014/539410.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1155/2014/539410}} An Overview of FastFlow: Combining Pattern-Level Abstraction and Efficiency in GPGPUs
Marco Aldinucci, Massimo Torquati, Maurizio Drocco, Guilherme Peretti Pezzi, Concetto Spampinato
GPU Technology Conference (GTC)
fastflow HPC impact paraphrase
Get an overview of FastFlow's parallel patterns can be used to design parallel applications for execution on both CPUs and GPGPUs while avoiding most of the complex low-level detail needed to make them efficient, portable and rapid to prototype. For a more detailed and technical review of FastFlow's parallel patterns as well as a use case where we will show the design and effectiveness of a novel universal image filtering template based on the variational approach.
@inproceedings{ff:gtc:2014:short,
abstract = {Get an overview of FastFlow's parallel patterns can be used to design parallel applications for execution on both CPUs and GPGPUs while avoiding most of the complex low-level detail needed to make them efficient, portable and rapid to prototype. For a more detailed and technical review of FastFlow's parallel patterns as well as a use case where we will show the design and effectiveness of a novel universal image filtering template based on the variational approach.},
address = {San Jose, CA, USA},
author = {Aldinucci, Marco and Torquati, Massimo and Drocco, Maurizio and Peretti Pezzi, Guilherme and Spampinato, Concetto},
booktitle = {GPU Technology Conference (GTC)},
date-added = {2014-04-13 23:20:52 +0000},
date-modified = {2017-12-12 13:54:20 +0000},
keywords = {fastflow, HPC, impact, paraphrase},
month = mar,
title = {An Overview of FastFlow: Combining Pattern-Level Abstraction and Efficiency in {GPGPUs}},
url = {http://calvados.di.unipi.it/storage/talks/2014_S4585-Marco-Aldinucci.pdf},
year = {2014},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/talks/2014_S4585-Marco-Aldinucci.pdf}} Pool evolution: a domain specific parallel pattern
Marco Aldinucci, Sonia Campa, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
Proc.of the 7th Intl. Symposium on High-level Parallel Programming and Applications (HLPP)
fastflow paraphrase repara
We introduce a new parallel pattern derived from a specific application domain and show how it turns out to have application beyond its domain of origin. The pool evolution pattern models the parallel evolution of a population subject to mutations and evolving in such a way that a given fitness function is optimized. The pattern has been demonstrated to be suitable for capturing and modeling the parallel patterns underpinning various evolutionary algorithms, as well as other parallel patterns typical of symbolic computation. In this paper we introduce the pattern, developed in the framework of the ParaPhrase EU-funded FP7 project, we discuss its implementation on modern multi/many core architectures and finally present experimental results obtained with FastFlow and Erlang implementations to assess its feasibility and scalability.
@inproceedings{2014:ff:pool:hlpp,
abstract = {We introduce a new parallel pattern derived from a specific application domain and show how it turns out to have application beyond its domain of origin. The pool evolution pattern models the parallel evolution of a population subject to mutations and evolving in such a way that a given fitness function is optimized. The pattern has been demonstrated to be suitable for capturing and modeling the parallel patterns underpinning various evolutionary algorithms, as well as other parallel patterns typical of symbolic computation. In this paper we introduce the pattern, developed in the framework of the ParaPhrase EU-funded FP7 project, we discuss its implementation on modern multi/many core architectures and finally present experimental results obtained with FastFlow and Erlang implementations to assess its feasibility and scalability.},
address = {Amsterdam, The Netherlands},
author = {Marco Aldinucci and Sonia Campa and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
booktitle = {Proc.of the 7th Intl. Symposium on High-level Parallel Programming and Applications (HLPP)},
date-modified = {2015-09-27 12:14:30 +0000},
keywords = {fastflow, paraphrase, repara},
month = jul,
title = {Pool evolution: a domain specific parallel pattern},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_hlpp_pool.pdf},
year = {2014},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2014_hlpp_pool.pdf}} 2013
2013
Memory affinity in multi-threading: the Bowtie2 case study
Claudia Misale, Marco Aldinucci, Massimo Torquati
Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) – Poster Abstracts
fastflow
The diffusion of the Next Generation Sequencing (NGS) has increased the amount of data obtainable by genomic experiments. From a DNA sample a NGS run is able to produce millions of short sequences (called reads), which should be mapped into a reference genome. In this paper, we analyse the performance of Bowtie2, a fast and popular DNA mapping tool. Bowtie2 exhibits a multithreading implementation on top of pthreads, spin-locks and SSE2 SIMD extension. From parallel computing viewpoint, is a paradigmatic example of a software requiring to address three fundamental problems in shared-memory programming for cache-coherent multi-core platforms: synchronisation efficiency at very fine grain (due to short reads), load-balancing (due to long reads), and efficient usage of memory subsystem (due to SSE2 memory pressure). We compare the original implementation against an alternative implementation on top of the FastFlow pattern-based programming framework. The proposed design exploits the high-level farm pattern of FastFlow, which is implemented top of nonblocking multi-threading and lock-less (CAS-free) queues, and provides the programmer with high-level mechanism to tune task scheduling to achieve both load-balancing and memory affinity. The proposed design, despite the high-level design, is always faster and more scalable with respect to the original one. The design of both original and alternative version will be presented along with their experimental evaluation on real-world data sets.
@inproceedings{ff:acaces:13,
abstract = {The diffusion of the Next Generation Sequencing (NGS) has increased the amount of data obtainable by genomic experiments. From a DNA sample a NGS run is able to produce millions of short sequences (called reads), which should be mapped into a reference genome. In this paper, we analyse the performance of Bowtie2, a fast and popular DNA mapping tool. Bowtie2 exhibits a multithreading implementation on top of pthreads, spin-locks and SSE2 SIMD extension. From parallel computing viewpoint, is a paradigmatic example of a software requiring to address three fundamental problems in shared-memory programming for cache-coherent multi-core platforms: synchronisation efficiency at very fine grain (due to short reads), load-balancing (due to long reads), and efficient usage of memory subsystem (due to SSE2 memory pressure). We compare the original implementation against an alternative implementation on top of the FastFlow pattern-based programming framework. The proposed design exploits the high-level farm pattern of FastFlow, which is implemented top of nonblocking multi-threading and lock-less (CAS-free) queues, and provides the programmer with high-level mechanism to tune task scheduling to achieve both load-balancing and memory affinity. The proposed design, despite the high-level design, is always faster and more scalable with respect to the original one. The design of both original and alternative version will be presented along with their experimental evaluation on real-world data sets.},
address = {Fiuggi, Italy},
author = {Claudia Misale and Marco Aldinucci and Massimo Torquati},
booktitle = {Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) -- Poster Abstracts},
date-added = {2015-03-21 15:12:59 +0000},
date-modified = {2015-03-21 15:12:59 +0000},
isbn = {9789038221908},
keywords = {fastflow},
publisher = {HiPEAC},
title = {Memory affinity in multi-threading: the Bowtie2 case study},
url = {https://iris.unito.it/retrieve/handle/2318/143005/23874/2013_ACACES_ex-abstract.pdf},
year = {2013},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/143005/23874/2013_ACACES_ex-abstract.pdf}} Parallel stochastic simulators in systems biology: the evolution of the species
Maurizio Drocco
Computer Science Department, University of Torino, Italy
fastflow
The stochastic simulation of biological systems is an increasingly popular technique in bioinformatics. It is often an enlightening technique, especially for multi-stable systems whose dynamics can be hardly captured with ordinary differential equations. To be effective, stochastic simulations should be supported by powerful statistical analysis tools. The simulation/analysis workflow may however result in being computationally expensive, thus compromising the interactivity required especially in model tuning. In this work we discuss the main opportunities to speed up the framework by parallelisation on modern multicore and hybrid multicore and distributed platforms, advocating the high-level design of simulators for stochastic systems as a vehicle for building efficient and portable parallel simulators endowed with on-line statistical analysis. In particular, the Calculus of Wrapped Compartments (CWC) Simulator, which is designed according to the FastFlow's pattern-based approach, is presented and discussed in this work.
@mastersthesis{tesi:drocco:13,
abstract = {The stochastic simulation of biological systems is an increasingly popular technique in bioinformatics. It is often an enlightening technique, especially for multi-stable systems whose dynamics can be hardly captured with ordinary differential equations. To be effective, stochastic simulations should be supported by powerful statistical analysis tools. The simulation/analysis workflow may however result in being computationally expensive, thus compromising the interactivity required especially in model tuning. In this work we discuss the main opportunities to speed up the framework by parallelisation on modern multicore and hybrid multicore and distributed platforms, advocating the high-level design of simulators for stochastic systems as a vehicle for building efficient and portable parallel simulators endowed with on-line statistical analysis. In particular, the Calculus of Wrapped Compartments (CWC) Simulator, which is designed according to the FastFlow's pattern-based approach, is presented and discussed in this work.},
author = {Maurizio Drocco},
date-modified = {2013-11-24 00:29:54 +0000},
keywords = {fastflow},
month = jul,
school = {Computer Science Department, University of Torino, Italy},
title = {Parallel stochastic simulators in systems biology: the evolution of the species},
url = {http://calvados.di.unipi.it/storage/paper_files/2013_tesi_drocco.pdf},
year = {2013},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2013_tesi_drocco.pdf}} Structured Data Access Annotations for Massively Parallel Computations
Marco Aldinucci, Sonia Campa, Peter Kilpatrick, Massimo Torquati
Euro-Par 2012 Workshops, Proc. of the ParaPhrase Workshop on Parallel Processing
fastflow paraphrase
We describe an approach aimed at addressing the issue of joint exploitation of control (stream) and data parallelism in a skele-ton based parallel programming environment, based on annotations and refactoring. Annotations drive efficient implementation of a parallel com-putation. Refactoring is used to transform the associated skeleton tree into a more efficient, functionally equivalent skeleton tree. In most cases,cost models are used to drive the refactoring process. We show howsample use case applications/kernels may be optimized and discuss pre-liminary experiments with FastFlow assessing the theoretical results.
@inproceedings{annotation:para:12,
abstract = {We describe an approach aimed at addressing the issue of joint exploitation of control (stream) and data parallelism in a skele-ton based parallel programming environment, based on annotations and refactoring. Annotations drive efficient implementation of a parallel com-putation. Refactoring is used to transform the associated skeleton tree into a more efficient, functionally equivalent skeleton tree. In most cases,cost models are used to drive the refactoring process. We show howsample use case applications/kernels may be optimized and discuss pre-liminary experiments with FastFlow assessing the theoretical results.},
author = {Marco Aldinucci and Sonia Campa and Peter Kilpatrick and Massimo Torquati},
booktitle = {Euro-Par 2012 Workshops, Proc. of the ParaPhrase Workshop on Parallel Processing},
date-added = {2012-07-23 21:22:03 +0000},
date-modified = {2015-09-27 12:49:52 +0000},
doi = {10.1007/978-3-642-36949-0_42},
keywords = {fastflow, paraphrase},
pages = {381--390},
publisher = {Springer},
series = {LNCS},
title = {Structured Data Access Annotations for Massively Parallel Computations},
url = {http://calvados.di.unipi.it/storage/paper_files/2013_annot_europar_workshops.pdf},
volume = {7640},
year = {2013},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2013_annot_europar_workshops.pdf},
bdsk-url-2 = {https://doi.org/10.1007/978-3-642-36949-0_42}} Targeting Distributed Systems in FastFlow
Marco Aldinucci, Sonia Campa, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
Euro-Par 2012 Workshops, Proc. of the CoreGrid Workshop on Grids, Clouds and P2P Computing
fastflow paraphrase
FastFlow is a structured parallel programming framework targeting shared memory multi-core architectures. In this paper we introduce a FastFlow extension aimed at supporting a network of multi-core workstation as well. The extension supports the execution of FastFlow programs by coordinating – in a structured way – the fine grain parallel activities running on a single workstation. We discuss the design and the implementation of this extension presenting preliminary experimental results validating it on state-of-the-art networked multi-core nodes.
@inproceedings{ff:distr:cgs:12,
abstract = {FastFlow is a structured parallel programming framework targeting shared memory multi-core architectures. In this paper we introduce a FastFlow extension aimed at supporting a network of multi-core workstation as well. The extension supports the execution of FastFlow programs by coordinating -- in a structured way -- the fine grain parallel activities running on a single workstation. We discuss the design and the implementation of this extension presenting preliminary experimental results validating it on state-of-the-art networked multi-core nodes.},
author = {Marco Aldinucci and Sonia Campa and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
booktitle = {Euro-Par 2012 Workshops, Proc. of the CoreGrid Workshop on Grids, Clouds and P2P Computing},
date-added = {2012-07-23 21:22:03 +0000},
date-modified = {2015-09-27 12:47:54 +0000},
doi = {10.1007/978-3-642-36949-0_7},
keywords = {fastflow, paraphrase},
pages = {47--56},
publisher = {Springer},
series = {LNCS},
title = {Targeting Distributed Systems in FastFlow},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_distr_ff_cgsymph.pdf},
volume = {7640},
year = {2013},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2012_distr_ff_cgsymph.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-642-36949-0_7}} Parallel stochastic simulators in system biology: the evolution of the species
Marco Aldinucci, Fabio Tordini, Maurizio Drocco, Massimo Torquati, Mario Coppo
Proc. of 21st Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)
bioinformatics fastflow
The stochastic simulation of biological systems is an increasingly popular technique in Bioinformatics. It is often an enlightening technique, especially for multi-stable systems which dynamics can be hardly captured with ordinary differential equations. To be effective, stochastic simulations should be supported by powerful statistical analysis tools. The simulation-analysis workflow may however result in being computationally expensive, thus compromising the interactivity required in model tuning. In this work we advocate the high-level design of simulators for stochastic systems as a vehicle for building efficient and portable parallel simulators. In particular, the Calculus of Wrapped Components (CWC) simulator, which is designed according to the FastFlow's pattern-based approach, is presented and discussed in this work. FastFlow has been extended to support also clusters of multi-cores with minimal coding effort, assessing the portability of the approach.
@inproceedings{ff_cwc_distr:pdp:13,
abstract = {The stochastic simulation of biological systems is an increasingly popular technique in Bioinformatics. It is often an enlightening technique, especially for multi-stable systems which dynamics can be hardly captured with ordinary differential equations. To be effective, stochastic simulations should be supported by powerful statistical analysis tools. The simulation-analysis workflow may however result in being computationally expensive, thus compromising the interactivity required in model tuning. In this work we advocate the high-level design of simulators for stochastic systems as a vehicle for building efficient and portable parallel simulators. In particular, the Calculus of Wrapped Components (CWC) simulator, which is designed according to the FastFlow's pattern-based approach, is presented and discussed in this work. FastFlow has been extended to support also clusters of multi-cores with minimal coding effort, assessing the portability of the approach.},
address = {Belfast, Nothern Ireland, U.K.},
author = {Marco Aldinucci and Fabio Tordini and Maurizio Drocco and Massimo Torquati and Mario Coppo},
booktitle = {Proc. of 21st Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)},
date-added = {2012-01-20 19:22:15 +0100},
date-modified = {2017-12-12 13:53:10 +0000},
doi = {10.1109/PDP.2013.66},
keywords = {fastflow, bioinformatics},
month = feb,
publisher = {IEEE},
title = {Parallel stochastic simulators in system biology: the evolution of the species},
url = {http://calvados.di.unipi.it/storage/paper_files/2013_cwc_d_PDP.pdf},
year = {2013},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2013_cwc_d_PDP.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1109/PDP.2013.66}} An abstract annotation model for skeletons
Marco Aldinucci, Sonia Campa, Fabio Tordini, Massimo Torquati, Peter Kilpatrick
Formal Methods for Components and Objects: Intl. Symposium, FMCO 2011, Torino, Italy, October 3-5, 2011, Revised Invited Lectures
fastflow paraphrase
Multi-core and many-core platforms are becoming increasingly heterogeneous and asymmetric. This significantly increases the porting and tuning effort required for parallel codes, which in turn often leads to a growing gap between peak machine power and actual application performance. In this work a first step toward the automated optimization of high level skeleton-based parallel code is discussed. The paper presents an abstract annotation model for skeleton programs aimed at formally describing suitable mapping of parallel activities on a high-level platform representation. The derived mapping and scheduling strategies are used to generate optimized run-time code.
@incollection{toolchain:fmco:11,
abstract = {Multi-core and many-core platforms are becoming increasingly heterogeneous and asymmetric. This significantly increases the porting and tuning effort required for parallel codes, which in turn often leads to a growing gap between peak machine power and actual application performance. In this work a first step toward the automated optimization of high level skeleton-based parallel code is discussed. The paper presents an abstract annotation model for skeleton programs aimed at formally describing suitable mapping of parallel activities on a high-level platform representation. The derived mapping and scheduling strategies are used to generate optimized run-time code.},
author = {Marco Aldinucci and Sonia Campa and Fabio Tordini and Massimo Torquati and Peter Kilpatrick},
booktitle = {Formal Methods for Components and Objects: Intl. Symposium, FMCO 2011, Torino, Italy, October 3-5, 2011, Revised Invited Lectures},
date-added = {2012-06-04 19:23:25 +0200},
date-modified = {2013-11-24 00:33:41 +0000},
doi = {10.1007/978-3-642-35887-6_14},
editor = {Bernhard Beckert and Ferruccio Damiani and Frank S. de Boer and Marcello M. Bonsangue},
isbn = {978-3-642-35886-9},
keywords = {fastflow, paraphrase},
pages = {257--276},
publisher = {Springer},
series = {LNCS},
title = {An abstract annotation model for skeletons},
url = {http://calvados.di.unipi.it/storage/paper_files/2013_fmco11_annotation.pdf},
volume = {7542},
year = {2013},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2013_fmco11_annotation.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-642-35887-6_14}} 2012
2012
On Parallelizing On-Line Statistics for Stochastic Biological Simulations
Marco Aldinucci, Mario Coppo, Ferruccio Damiani, Maurizio Drocco, Eva Sciacca, Salvatore Spinella, Massimo Torquati, Angelo Troina
Proc. of Euro-Par Workshops: 2nd Workshop on High Performance Bioinformatics and Biomedicine (HiBB)
bioinformatics fastflow
This work concerns a general technique to enrich parallel version of stochastic simulators for biological systems with tools for on-line statistical analysis of the results. In particular, within the FastFlow parallel programming framework, we describe the methodology and the implementation of a parallel Monte Carlo simulation infrastructure extended with user-defined on-line data filtering and mining functions. The simulator and the on-line analysis were validated on large multi-core platforms and representative proof-of-concept biological systems.
@inproceedings{cwcsim:onlinestats:ff:hibb:11,
abstract = {This work concerns a general technique to enrich parallel version of stochastic simulators for biological systems with tools for on-line statistical analysis of the results. In particular, within the FastFlow parallel programming framework, we describe the methodology and the implementation of a parallel Monte Carlo simulation infrastructure extended with user-defined on-line data filtering and mining functions. The simulator and the on-line analysis were validated on large multi-core platforms and representative proof-of-concept biological systems.},
address = {Bordeaux, France},
author = {Marco Aldinucci and Mario Coppo and Ferruccio Damiani and Maurizio Drocco and Eva Sciacca and Salvatore Spinella and Massimo Torquati and Angelo Troina},
booktitle = {Proc. of Euro-Par Workshops: 2nd Workshop on High Performance Bioinformatics and Biomedicine (HiBB)},
date-added = {2010-08-15 00:50:09 +0200},
date-modified = {2017-12-12 14:47:15 +0000},
doi = {10.1007/978-3-642-29740-3_2},
editor = {Michael Alexander and Pasqua D'Ambra and Adam Belloum and George Bosilca and Mario Cannataro and Marco Danelutto and Beniamino Di Martino and Michael Gerndt and Emmanuel Jeannot and Raymond Namyst and Jean Roman and Stephen L. Scott and Jesper Larsson Tr{\"a}ff and Geoffroy Vall{\'e}e and Josef Weidendorfer},
keywords = {bioinformatics, fastflow},
pages = {3--12},
publisher = {Springer},
series = {LNCS},
title = {On Parallelizing On-Line Statistics for Stochastic Biological Simulations},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_onlinestat_HiBB2011.pdf},
volume = {7156},
year = {2012},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2012_onlinestat_HiBB2011.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-642-29740-3_2}} Parallel patterns + Macro Data Flow for multi-core programming
Marco Aldinucci, Marco Danelutto, Lorenzo Anardu, Massimo Torquati, Peter Kilpatrick
Proc. of Intl. Euromicro PDP 2012: Parallel Distributed and network-based Processing
fastflow
Data flow techniques have been around since the early '70s when they were used in compilers for sequential languages. Shortly after their introduction they were also considered as a possible model for parallel computing, although the impact here was limited. Recently, however, data flow has been identified as a candidate for efficient implementation of various programming models on multi-core architectures. In most cases, however, the burden of determining data flow “macro” instructions is left to the programmer, while the compiler/run time system manages only the efficient scheduling of these instructions. We discuss a structured parallel programming approach supporting automatic compilation of programs to macro data flow and we show experimental results demonstrating the feasibility of the approach and the efficiency of the resulting “object” code on different classes of state-of-the-art multi-core architectures. The experimental results use different base mechanisms to implement the macro data flow run time support, from plain pthreads with condition variables to more modern and effective lock- and fence-free parallel frameworks. Experimental results comparing efficiency of the proposed approach with those achieved using other, more classical, parallel frameworks are also presented.
@inproceedings{dataflow:pdp:12,
abstract = {Data flow techniques have been around since the early '70s when they were used in compilers for sequential languages. Shortly after their introduction they were also considered as a possible model for parallel computing, although the impact here was limited. Recently, however, data flow has been identified as a candidate for efficient implementation of various programming models on multi-core architectures. In most cases, however, the burden of determining data flow ``macro'' instructions is left to the programmer, while the compiler/run time system manages only the efficient scheduling of these instructions. We discuss a structured parallel programming approach supporting automatic compilation of programs to macro data flow and we show experimental results demonstrating the feasibility of the approach and the efficiency of the resulting ``object'' code on different classes of state-of-the-art multi-core architectures. The experimental results use different base mechanisms to implement the macro data flow run time support, from plain pthreads with condition variables to more modern and effective lock- and fence-free parallel frameworks. Experimental results comparing efficiency of the proposed approach with those achieved using other, more classical, parallel frameworks are also presented.},
address = {Garching, Germany},
author = {Marco Aldinucci and Marco Danelutto and Lorenzo Anardu and Massimo Torquati and Peter Kilpatrick},
booktitle = {Proc. of Intl. Euromicro PDP 2012: Parallel Distributed and network-based Processing},
date-added = {2012-10-24 17:29:14 +0000},
date-modified = {2013-11-24 00:35:34 +0000},
doi = {10.1109/PDP.2012.44},
keywords = {fastflow},
month = feb,
pages = {27--36},
publisher = {IEEE},
title = {Parallel patterns + Macro Data Flow for multi-core programming},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_mdf_PDP.pdf},
year = {2012},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2012_mdf_PDP.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1109/PDP.2012.44}} A Parallel Edge Preserving Algorithm for Salt and Pepper Image Denoising
Marco Aldinucci, Concetto Spampinato, Maurizio Drocco, Massimo Torquati, Simone Palazzo
Proc. of 2nd Intl. Conference on Image Processing Theory Tools and Applications (IPTA)
fastflow impact
In this paper a two-phase filter for removing “salt and pepper” noise is proposed. In the first phase, an adaptive median filter is used to identify the set of the noisy pixels; in the second phase, these pixels are restored according to a regularization method, which contains a data-fidelity term reflecting the impulse noise characteristics. The algorithm, which exhibits good performance both in denoising and in restoration, can be easily and effectively parallelized to exploit the full power of multi-core CPUs and GPGPUs; the proposed implementation based on the FastFlow library achieves both close-to-ideal speedup and very good wall-clock execution figures.
@inproceedings{denoiser:ff:ipta:12,
abstract = {In this paper a two-phase filter for removing ``salt and pepper'' noise is proposed. In the first phase, an adaptive median filter is used to identify the set of the noisy pixels; in the second phase, these pixels are restored according to a regularization method, which contains a data-fidelity term reflecting the impulse noise characteristics. The algorithm, which exhibits good performance both in denoising and in restoration, can be easily and effectively parallelized to exploit the full power of multi-core CPUs and GPGPUs; the proposed implementation based on the FastFlow library achieves both close-to-ideal speedup and very good wall-clock execution figures.},
address = {Istambul, Turkey},
author = {Marco Aldinucci and Concetto Spampinato and Maurizio Drocco and Massimo Torquati and Simone Palazzo},
booktitle = {Proc. of 2nd Intl. Conference on Image Processing Theory Tools and Applications (IPTA)},
date-added = {2012-06-04 18:38:01 +0200},
date-modified = {2015-09-27 12:53:53 +0000},
doi = {10.1109/IPTA.2012.6469567},
editor = {K. Djemal and M. Deriche and W. Puech and Osman N. Ucan},
isbn = {978-1-4673-2582-0},
keywords = {fastflow, impact},
month = oct,
pages = {97--102},
publisher = {IEEE},
title = {A Parallel Edge Preserving Algorithm for Salt and Pepper Image Denoising},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_2phasedenoiser_ff_ipta.pdf},
year = {2012},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2012_2phasedenoiser_ff_ipta.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1109/IPTA.2012.6469567}} FastFlow tutorial
Marco Aldinucci, Marco Danelutto, Massimo Torquati
fastflow
@techreport{fastflow_tutorial:TR-12-04:12,
author = {Marco Aldinucci and Marco Danelutto and Massimo Torquati},
date-added = {2011-03-17 23:19:05 +0100},
date-modified = {2013-11-24 00:34:55 +0000},
institution = {Universit\`a di Pisa, Dipartimento di Informatica, Italy},
keywords = {fastflow},
month = mar,
number = {TR-12-04},
title = {FastFlow tutorial},
url = {http://compass2.di.unipi.it/TR/Files/TR-12-04.pdf.gz},
year = {2012},
bdsk-url-1 = {http://compass2.di.unipi.it/TR/Files/TR-12-04.pdf.gz}} High-level lock-less programming for multicore
Fabio Tordini, Marco Aldinucci, Massimo Torquati
Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) – Poster Abstracts
fastflow
Modern computers are built upon multi-core architectures. Achieving peak performance on these architectures is hard and may require a substantial programming effort. The synchronisation of many processes racing to access a common resource (the shared memory) has been a fundamental problem on parallel computing for years, and many solutions have been proposed to address this issue. Non-blocking synchronisation and transactional primitives have been envisioned as a way to reduce memory wall problem. Despite sometimes effective (and exhibiting a great momentum in the research community), they are only one facet of the problem, as their exploitation still requires non-trivial programming skills. With non-blocking philosophy in mind, we propose high-level programming patterns that will relieve the programmer from worrying about low-level details such as synchronisation of racing processes as well as those fine tunings needed to improve the overall performance, like proper (distributed) dynamic memory allocation and effective exploitation of the memory hierarchy.
@inproceedings{ff:acaces:12,
abstract = {Modern computers are built upon multi-core architectures. Achieving peak performance on these architectures is hard and may require a substantial programming effort. The synchronisation of many processes racing to access a common resource (the shared memory) has been a fundamental problem on parallel computing for years, and many solutions have been proposed to address this issue. Non-blocking synchronisation and transactional primitives have been envisioned as a way to reduce memory wall problem. Despite sometimes effective (and exhibiting a great momentum in the research community), they are only one facet of the problem, as their exploitation still requires non-trivial programming skills. With non-blocking philosophy in mind, we propose high-level programming patterns that will relieve the programmer from worrying about low-level details such as synchronisation of racing processes as well as those fine tunings needed to improve the overall performance, like proper (distributed) dynamic memory allocation and effective exploitation of the memory hierarchy.},
address = {Fiuggi, Italy},
author = {Fabio Tordini and Marco Aldinucci and Massimo Torquati},
booktitle = {Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) -- Poster Abstracts},
date-added = {2012-07-17 17:58:06 +0200},
date-modified = {2013-11-24 00:36:10 +0000},
isbn = {9789038219875},
keywords = {fastflow},
publisher = {HiPEAC},
title = {High-level lock-less programming for multicore},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_ACACES_ex-abstract.pdf},
year = {2012},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2012_ACACES_ex-abstract.pdf}} An Efficient Unbounded Lock-Free Queue for Multi-core Systems
Marco Aldinucci, Marco Danelutto, Peter Kilpatrick, Massimiliano Meneghin, Massimo Torquati
Proc. of 18th Intl. Euro-Par 2012 Parallel Processing
fastflow paraphrase
The use of efficient synchronization mechanisms is crucial for implementing fine grained parallel programs on modern shared cache multi-core architectures. In this paper we study this problem by considering Single-Producer/Single-Consumer (SPSC) coordination using unbounded queues. A novel unbounded SPSC algorithm capable of reducing the row synchronization latency and speeding up Producer-Consumer coordination is presented. The algorithm has been extensively tested on a shared-cache multi-core platform and a sketch proof of correctness is presented. The queues proposed have been used as basic building blocks to implement the FastFlow parallel framework, which has been demonstrated to offer very good performance for fine-grain parallel applications.
@inproceedings{ff:spsc:europar:12,
abstract = {The use of efficient synchronization mechanisms is crucial for implementing fine grained parallel programs on modern shared cache multi-core architectures. In this paper we study this problem by considering Single-Producer/Single-Consumer (SPSC) coordination using unbounded queues. A novel unbounded SPSC algorithm capable of reducing the row synchronization latency and speeding up Producer-Consumer coordination is presented. The algorithm has been extensively tested on a shared-cache multi-core platform and a sketch proof of correctness is presented. The queues proposed have been used as basic building blocks to implement the FastFlow parallel framework, which has been demonstrated to offer very good performance for fine-grain parallel applications.},
address = {Rhodes Island, Greece},
author = {Marco Aldinucci and Marco Danelutto and Peter Kilpatrick and Massimiliano Meneghin and Massimo Torquati},
booktitle = {Proc. of 18th Intl. Euro-Par 2012 Parallel Processing},
date-added = {2011-04-19 10:22:00 +0200},
date-modified = {2015-09-27 12:55:20 +0000},
doi = {10.1007/978-3-642-32820-6_65},
keywords = {fastflow, paraphrase},
month = aug,
pages = {662--673},
publisher = {Springer},
series = {LNCS},
title = {An Efficient Unbounded Lock-Free Queue for Multi-core Systems},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_spsc_europar.pdf},
volume = {7484},
year = {2012},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2012_spsc_europar.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-642-32820-6_65}} Targeting heterogeneous architectures via macro data flow
Marco Aldinucci, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
Parallel Processing Letters
fastflow paraphrase
We propose a data flow based run time system as an efficient tool for supporting execution of parallel code on heterogeneous architectures hosting both multicore CPUs and GPUs. We discuss how the proposed run time system may be the target of both structured parallel applications developed using algorithmic skeletons/parallel design patterns and also more “domain specific” programming models. Experimental results demonstrating the feasibility of the approach are presented.
@article{mdf:hplgpu:ppl:12,
abstract = {We propose a data flow based run time system as an efficient tool for supporting execution of parallel code on heterogeneous architectures hosting both multicore CPUs and GPUs. We discuss how the proposed run time system may be the target of both structured parallel applications developed using algorithmic skeletons/parallel design patterns and also more ``domain specific'' programming models. Experimental results demonstrating the feasibility of the approach are presented.},
annote = {Extended version of Intl. Workshop on High-level Programming for Heterogeneous and Hierarchical Parallel Systems (HLPGPU)},
author = {Marco Aldinucci and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
date-added = {2012-04-25 13:20:40 +0000},
date-modified = {2015-09-27 12:55:11 +0000},
doi = {10.1142/S0129626412400063},
issn = {0129-6264},
journal = {Parallel Processing Letters},
keywords = {fastflow, paraphrase},
month = jun,
number = {2},
title = {Targeting heterogeneous architectures via macro data flow},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_mdf_PPL-hplgpu.pdf},
volume = {22},
year = {2012},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2012_mdf_PPL-hplgpu.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1142/S0129626412400063}} 2011
2011
Accelerating code on multi-cores with FastFlow
Marco Aldinucci, Marco Danelutto, Peter Kilpatrick, Massimiliano Meneghin, Massimo Torquati
Proc. of 17th Intl. Euro-Par 2011 Parallel Processing
fastflow
FastFlow is a programming framework specifically targeting cache-coherent shared-memory multicores. It is implemented as a stack of C++ template libraries built on top of lock-free (and memory fence free) synchronization mechanisms. Its philosophy is to combine programmability with performance. In this paper a new FastFlow programming methodology aimed at supporting parallelization of existing sequential code via offloading onto a dynamically created software accelerator is presented. The new methodology has been validated using a set of simple micro-benchmarks and some real applications.
@inproceedings{ff:acc:europar:11,
abstract = {FastFlow is a programming framework specifically targeting cache-coherent shared-memory multicores. It is implemented as a stack of C++ template libraries built on top of lock-free (and memory fence free) synchronization mechanisms. Its philosophy is to combine programmability with performance. In this paper a new FastFlow programming methodology aimed at supporting parallelization of existing sequential code via offloading onto a dynamically created software accelerator is presented. The new methodology has been validated using a set of simple micro-benchmarks and some real applications.},
address = {Bordeaux, France},
author = {Marco Aldinucci and Marco Danelutto and Peter Kilpatrick and Massimiliano Meneghin and Massimo Torquati},
booktitle = {Proc. of 17th Intl. Euro-Par 2011 Parallel Processing},
date-added = {2012-06-04 18:35:57 +0200},
date-modified = {2013-12-12 00:46:59 +0000},
doi = {10.1007/978-3-642-23397-5_17},
editor = {E. Jeannot and R. Namyst and J. Roman},
keywords = {fastflow},
month = aug,
pages = {170--181},
publisher = {Springer},
series = {LNCS},
title = {Accelerating code on multi-cores with FastFlow},
url = {http://calvados.di.unipi.it/storage/paper_files/2011_fastflow_acc_europar.pdf},
volume = {6853},
year = {2011},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2011_fastflow_acc_europar.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-642-23397-5_17}} On Designing Multicore-Aware Simulators for Biological Systems
Marco Aldinucci, Mario Coppo, Ferruccio Damiani, Maurizio Drocco, Massimo Torquati, Angelo Troina
Proc. of 19th Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)
fastflow
The stochastic simulation of biological systems is an increasingly popular technique in bioinformatics. It often is an enlightening technique, which may however result in being computational expensive. We discuss the main opportunities to speed it up on multi-core platforms, which pose new challenges for parallelisation techniques. These opportunities are developed in two general families of solutions involving both the single simulation and a bulk of independent simulations (either replicas of derived from parameter sweep). Proposed solutions are tested on the parallelisation of the CWC simulator (Calculus of Wrapped Compartments) that is carried out according to proposed solutions by way of the FastFlow programming framework making possible fast development and efficient execution on multi-cores.
@inproceedings{ff:cwc:pdp:11,
abstract = {The stochastic simulation of biological systems is an increasingly popular technique in bioinformatics. It often is an enlightening technique, which may however result in being computational expensive. We discuss the main opportunities to speed it up on multi-core platforms, which pose new challenges for parallelisation techniques. These opportunities are developed in two general families of solutions involving both the single simulation and a bulk of independent simulations (either replicas of derived from parameter sweep). Proposed solutions are tested on the parallelisation of the CWC simulator (Calculus of Wrapped Compartments) that is carried out according to proposed solutions by way of the FastFlow programming framework making possible fast development and efficient execution on multi-cores.},
address = {Ayia Napa, Cyprus},
author = {Marco Aldinucci and Mario Coppo and Ferruccio Damiani and Maurizio Drocco and Massimo Torquati and Angelo Troina},
booktitle = {Proc. of 19th Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)},
date-added = {2012-02-25 01:21:25 +0000},
date-modified = {2017-12-12 13:51:21 +0000},
doi = {10.1109/PDP.2011.81},
editor = {Yiannis Cotronis and Marco Danelutto and George Angelos Papadopoulos},
keywords = {fastflow},
month = feb,
pages = {318--325},
publisher = {IEEE},
title = {On Designing Multicore-Aware Simulators for Biological Systems},
url = {http://calvados.di.unipi.it/storage/paper_files/2011_ff_cwc_sim_PDP.pdf},
year = {2011},
bdsk-url-1 = {http://arxiv.org/pdf/1010.2438v2},
bdsk-url-2 = {http://calvados.di.unipi.it/storage/paper_files/2011_ff_cwc_sim_PDP.pdf},
bdsk-url-3 = {http://dx.doi.org/10.1109/PDP.2011.81}} Porting Decision Tree Building and Pruning Algorithms to Multicore using FastFlow
Marco Aldinucci, Salvatore Ruggieri, Massimo Torquati
fastflow
@techreport{TR-11-06,
author = {Marco Aldinucci and Salvatore Ruggieri and Massimo Torquati},
date-added = {2012-04-15 18:40:07 +0000},
date-modified = {2013-11-24 00:37:04 +0000},
institution = {Universit\`a di Pisa, Dipartimento di Informatica, Italy},
keywords = {fastflow},
month = mar,
number = {TR-11-06},
title = {Porting Decision Tree Building and Pruning Algorithms to Multicore using FastFlow},
url = {http://compass2.di.unipi.it/TR/Files/TR-11-06.pdf.gz},
year = {2011},
bdsk-url-1 = {http://compass2.di.unipi.it/TR/Files/TR-11-06.pdf.gz}} 2010
2010
Efficient Smith-Waterman on multi-core with FastFlow
Marco Aldinucci, Massimiliano Meneghin, Massimo Torquati
Proc. of Intl. Euromicro PDP 2010: Parallel Distributed and network-based Processing
fastflow
Shared memory multiprocessors have returned to popularity thanks to rapid spreading of commodity multi-core architectures. However, little attention has been paid to supporting effective streaming applications on these architectures. In this paper we describe FastFlow, a low-level programming framework based on lock-free queues explicitly designed to support high-level languages for streaming applications. We compare FastFlow with state-of-the-art programming frameworks such as Cilk, OpenMP, and Intel TBB. We experimentally demonstrate that FastFlow is always more efficient than them on a given real world application: the speedup of FastFlow over other solutions may be substantial for fine grain tasks, for example +35
@inproceedings{fastflow:pdp:10,
abstract = {Shared memory multiprocessors have returned to popularity thanks to rapid spreading of commodity multi-core architectures. However, little attention has been paid to supporting effective streaming applications on these architectures. In this paper we describe FastFlow, a low-level programming framework based on lock-free queues explicitly designed to support high-level languages for streaming applications. We compare FastFlow with state-of-the-art programming frameworks such as Cilk, OpenMP, and Intel TBB. We experimentally demonstrate that FastFlow is always more efficient than them on a given real world application: the speedup of FastFlow over other solutions may be substantial for fine grain tasks, for example +35% over OpenMP, +226% over Cilk, +96% over TBB for the alignment of protein P01111 against UniProt DB using the Smith-Waterman algorithm.},
address = {Pisa, Italy},
author = {Marco Aldinucci and Massimiliano Meneghin and Massimo Torquati},
booktitle = {Proc. of Intl. Euromicro PDP 2010: Parallel Distributed and network-based Processing},
date-added = {2007-10-26 01:02:32 +0200},
date-modified = {2013-11-24 00:38:51 +0000},
doi = {10.1109/PDP.2010.93},
editor = {Marco Danelutto and Tom Gross and Julien Bourgeois},
keywords = {fastflow},
month = feb,
pages = {195--199},
publisher = {IEEE},
title = {Efficient {Smith-Waterman} on multi-core with FastFlow},
url = {http://calvados.di.unipi.it/storage/paper_files/2010_fastflow_SW_PDP.pdf},
year = {2010},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2010_fastflow_SW_PDP.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1109/PDP.2010.93}} Accelerating sequential programs using FastFlow and self-offloading
Marco Aldinucci, Marco Danelutto, Peter Kilpatrick, Massimiliano Meneghin, Massimo Torquati
fastflow
Shared memory multiprocessors come back to popularity thanks to rapid spreading of commodity multi-core architectures. As ever, shared memory programs are fairly easy to write and quite hard to optimise; providing multi-core programmers with optimising tools and programming frameworks is a nowadays challenge. Few efforts have been done to support effective streaming applications on these architectures. In this paper we introduce FastFlow, a low-level programming framework based on lock-free queues explicitly designed to support high-level languages for streaming applications. We compare FastFlow with state-of-the-art programming frameworks such as Cilk, OpenMP, and Intel TBB. We experimentally demonstrate that FastFlow is always more efficient than all of them in a set of micro-benchmarks and on a real world application; the speedup edge of FastFlow over other solutions might be bold for fine grain tasks, as an example +35
@techreport{fastflow_acc:tr-10-03,
abstract = {Shared memory multiprocessors come back to popularity thanks to rapid spreading of commodity multi-core architectures. As ever, shared memory programs are fairly easy to write and quite hard to optimise; providing multi-core programmers with optimising tools and programming frameworks is a nowadays challenge. Few efforts have been done to support effective streaming applications on these architectures. In this paper we introduce FastFlow, a low-level programming framework based on lock-free queues explicitly designed to support high-level languages for streaming applications. We compare FastFlow with state-of-the-art programming frameworks such as Cilk, OpenMP, and Intel TBB. We experimentally demonstrate that FastFlow is always more efficient than all of them in a set of micro-benchmarks and on a real world application; the speedup edge of FastFlow over other solutions might be bold for fine grain tasks, as an example +35% on OpenMP, +226% on Cilk, +96% on TBB for the alignment of protein P01111 against UniProt DB using Smith-Waterman algorithm.},
author = {Marco Aldinucci and Marco Danelutto and Peter Kilpatrick and Massimiliano Meneghin and Massimo Torquati},
date-added = {2009-09-08 16:14:34 +0200},
date-modified = {2013-11-24 00:39:01 +0000},
institution = {Universit{\`a} di Pisa, Dipartimento di Informatica, Italy},
keywords = {fastflow},
month = feb,
number = {TR-10-03},
title = {Accelerating sequential programs using {FastFlow} and self-offloading},
year = {2010},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/TR-09-12.pdf}} Porting Decision Tree Algorithms to Multicore using FastFlow
Marco Aldinucci, Salvatore Ruggieri, Massimo Torquati
Proc. of European Conference in Machine Learning and Knowledge Discovery in Databases (ECML PKDD)
fastflow
The whole computer hardware industry embraced multicores. For these machines, the extreme optimisation of sequential algorithms is no longer sufficient to squeeze the real machine power, which can be only exploited via thread-level parallelism. Decision tree algorithms exhibit natural concurrency that makes them suitable to be parallelised. This paper presents an approach for easy-yet-efficient porting of an implementation of the C4.5 algorithm on multicores. The parallel porting requires minimal changes to the original sequential code, and it is able to exploit up to 7X speedup on an Intel dual-quad core machine.
@inproceedings{fastflow_c45:emclpkdd,
abstract = {The whole computer hardware industry embraced multicores. For these machines, the extreme optimisation of sequential algorithms is no longer sufficient to squeeze the real machine power, which can be only exploited via thread-level parallelism. Decision tree algorithms exhibit natural concurrency that makes them suitable to be parallelised. This paper presents an approach for easy-yet-efficient porting of an implementation of the C4.5 algorithm on multicores. The parallel porting requires minimal changes to the original sequential code, and it is able to exploit up to 7X speedup on an Intel dual-quad core machine.},
address = {Barcelona, Spain},
author = {Marco Aldinucci and Salvatore Ruggieri and Massimo Torquati},
booktitle = {Proc. of European Conference in Machine Learning and Knowledge Discovery in Databases (ECML PKDD)},
date-added = {2010-06-15 21:03:56 +0200},
date-modified = {2013-11-24 00:38:07 +0000},
doi = {10.1007/978-3-642-15880-3_7},
editor = {Jos{\'e} L. Balc{\'a}zar and Francesco Bonchi and Aristides Gionis and Mich{\`e}le Sebag},
keywords = {fastflow},
month = sep,
pages = {7--23},
publisher = {Springer},
series = {LNCS},
title = {Porting Decision Tree Algorithms to Multicore using {FastFlow}},
url = {http://calvados.di.unipi.it/storage/paper_files/2010_c45FF_ECMLPKDD.pdf},
volume = {6321},
year = {2010},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2010_c45FF_ECMLPKDD.pdf},
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-642-15880-3_7}} Efficient Parallel MonteCarlo with FastFlow
Marco Aldinucci
HPC-Europa2: Science and Supercomputing in Europe, research highlights 2010
bioinformatics fastflow
The stochastic simulation of natural systems is a very informative but happens be computationally expensive. We present StochKit-FF, an parallel version of StochKit, a reference toolkit for stochastic simulations that sustantially improves StochKit performances on multi-core platforms.
@incollection{ff:hpc-europa:10,
abstract = {The stochastic simulation of natural systems is a very informative but happens be computationally expensive. We present StochKit-FF, an parallel version of StochKit, a reference toolkit for stochastic simulations that sustantially improves StochKit performances on multi-core platforms.},
author = {Marco Aldinucci},
booktitle = {HPC-Europa2: Science and Supercomputing in Europe, research highlights 2010},
date-added = {2011-06-18 18:43:19 +0200},
date-modified = {2013-11-24 00:40:04 +0000},
keywords = {bioinformatics, fastflow},
publisher = {Cineca},
title = {Efficient Parallel {MonteCarlo} with {FastFlow}},
url = {http://calvados.di.unipi.it/storage/paper_files/2010-ff_hpceuropa2_092-inform-Aldinucci.pdf},
year = {2010},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/2010-ff_hpceuropa2_092-inform-Aldinucci.pdf}} Formal Synthetic Immunology
Marco Aldinucci, Andrea Bracciali, Pietro Liò
ERCIM News
bioinformatics fastflow
The human immune system fights pathogens using an articulated set of strategies whose function is to maintain in health the organism. A large effort to formally model such a complex system using a computational approach is currently underway, with the goal of developing a discipline for engineering "synthetic" immune responses. This requires the integration of a range of analysis techniques developed for formally reasoning about the behaviour of complex dynamical systems. Furthermore, a novel class of software tools has to be developed, capable of efficiently analysing these systems on widely accessible computing platforms, such as commodity multi-core architectures..
@article{stochkitff:ercimnews:10,
abstract = {The human immune system fights pathogens using an articulated set of strategies whose function is to maintain in health the organism. A large effort to formally model such a complex system using a computational approach is currently underway, with the goal of developing a discipline for engineering "synthetic" immune responses. This requires the integration of a range of analysis techniques developed for formally reasoning about the behaviour of complex dynamical systems. Furthermore, a novel class of software tools has to be developed, capable of efficiently analysing these systems on widely accessible computing platforms, such as commodity multi-core architectures..},
author = {Marco Aldinucci and Andrea Bracciali and Pietro Li{\`o}},
date-added = {2010-07-02 20:32:31 +0200},
date-modified = {2013-11-24 00:38:19 +0000},
issn = {0926-4981},
journal = {ERCIM News},
keywords = {bioinformatics, fastflow},
month = jul,
pages = {40--41},
title = {Formal Synthetic Immunology},
url = {http://ercim-news.ercim.eu/images/stories/EN82/EN82-web.pdf},
volume = {82},
year = {2010},
bdsk-url-1 = {http://ercim-news.ercim.eu/images/stories/EN82/EN82-web.pdf}} Efficient streaming applications on multi-core with FastFlow: The biosequence alignment test-bed
Marco Aldinucci, Marco Danelutto, Massimiliano Meneghin, Massimo Torquati, Peter Kilpatrick
Elsevier
fastflow
Shared-memory multi-core architectures are becoming increasingly popular. While their parallelism and peak performance is ever increasing, their efficiency is often disappointing due to memory fence overheads. In this paper we present FastFlow, a programming methodology based on lock-free queues explicitly designed for programming streaming applications on multi-cores. The potential of FastFlow is evaluated on micro-benchmarks and on the Smith-Waterman sequence alignment application, which exhibits a substantial speedup against the state-of-the-art multi-threaded implementation (SWPS3 x86/SSE2).
@inbook{fastflow:parco:09,
abstract = {Shared-memory multi-core architectures are becoming increasingly popular. While their parallelism and peak performance is ever increasing, their efficiency is often disappointing due to memory fence overheads. In this paper we present FastFlow, a programming methodology based on lock-free queues explicitly designed for programming streaming applications on multi-cores. The potential of FastFlow is evaluated on micro-benchmarks and on the Smith-Waterman sequence alignment application, which exhibits a substantial speedup against the state-of-the-art multi-threaded implementation (SWPS3 x86/SSE2).},
author = {Aldinucci, Marco and Danelutto, Marco and Meneghin, Massimiliano and Torquati, Massimo and Kilpatrick, Peter},
date-modified = {2019-03-25 23:17:15 +0100},
doi = {10.3233/978-1-60750-530-3-273},
keywords = {fastflow},
language = {English},
opteditor = {Barbara Chapman and Fr{\'e}d{\'e}ric Desprez and Gerhard R. Joubert and Alain Lichnewsky and Frans Peters and Thierry Priol},
pages = {273--280},
publisher = {Elsevier},
series = {Advances in Parallel Computing},
title = {Efficient streaming applications on multi-core with FastFlow: The biosequence alignment test-bed},
url = {https://iris.unito.it/retrieve/handle/2318/67254/690859/2009_fastflow_parco.pdf},
volume = {19},
year = {2010},
bdsk-url-1 = {https://iris.unito.it/retrieve/handle/2318/67254/690859/2009_fastflow_parco.pdf},
bdsk-url-2 = {https://doi.org/10.3233/978-1-60750-530-3-273}} 2009
2009
FastFlow: Efficient Parallel Streaming Applications on Multi-core
Marco Aldinucci, Massimo Torquati, Massimiliano Meneghin
fastflow
Shared memory multiprocessors come back to popularity thanks to rapid spreading of commodity multi-core architectures. As ever, shared memory programs are fairly easy to write and quite hard to optimise; providing multi-core programmers with optimising tools and programming frameworks is a nowadays challenge. Few efforts have been done to support effective streaming applications on these architectures. In this paper we introduce FastFlow, a low-level programming framework based on lock-free queues explicitly designed to support high-level languages for streaming applications. We compare FastFlow with state-of-the-art programming frameworks such as Cilk, OpenMP, and Intel TBB. We experimentally demonstrate that FastFlow is always more efficient than all of them in a set of micro-benchmarks and on a real world application; the speedup edge of FastFlow over other solutions might be bold for fine grain tasks, as an example +35
@techreport{fastflow:tr-09-12,
abstract = {Shared memory multiprocessors come back to popularity thanks to rapid spreading of commodity multi-core architectures. As ever, shared memory programs are fairly easy to write and quite hard to optimise; providing multi-core programmers with optimising tools and programming frameworks is a nowadays challenge. Few efforts have been done to support effective streaming applications on these architectures. In this paper we introduce FastFlow, a low-level programming framework based on lock-free queues explicitly designed to support high-level languages for streaming applications. We compare FastFlow with state-of-the-art programming frameworks such as Cilk, OpenMP, and Intel TBB. We experimentally demonstrate that FastFlow is always more efficient than all of them in a set of micro-benchmarks and on a real world application; the speedup edge of FastFlow over other solutions might be bold for fine grain tasks, as an example +35% on OpenMP, +226% on Cilk, +96% on TBB for the alignment of protein P01111 against UniProt DB using Smith-Waterman algorithm.},
author = {Marco Aldinucci and Massimo Torquati and Massimiliano Meneghin},
date-added = {2010-02-13 16:20:18 +0100},
date-modified = {2013-11-24 00:39:38 +0000},
institution = {Universit{\`a} di Pisa, Dipartimento di Informatica, Italy},
keywords = {fastflow},
month = sep,
number = {TR-09-12},
title = {{FastFlow}: Efficient Parallel Streaming Applications on Multi-core},
url = {http://arxiv.org/abs/0909.1187},
year = {2009},
bdsk-url-1 = {http://calvados.di.unipi.it/storage/paper_files/TR-09-12.pdf},
bdsk-url-2 = {http://arxiv.org/abs/0909.1187}}Discontinued Parallel Computing tools
GAM: Global Asynchronous Memory
Parallel Programming with Global Asynchronous Memory: Models, C++ APIs and Implementations.
M. Drocco, “Parallel programming with global asynchronous memory: models, C++ APIs and implementations,” PhD Thesis, 2017. doi:10.5281/zenodo.1037585
PiCo: Pipeline Composition
PiCo (Pipeline Composition) is an open-source C++11 header-only DSL for high-performance data analytics, featuring low latency, high throughput, and minimal memory footprint on multi-core platforms. For more information see the PiCo paper.
GridCOMP Grid Component Model
The full software package supporting the development of distributed and multi-core applications based on autonomic components and behavioural skeletons is available under GPL license. More information on the GridCOMP page. The Grid Component Model (GCM) has been standardised by ETSI:
- DTS/GRID-0004-1 (27/08/2008)
- DTS/GRID-004-2 (27/08/2008)
- DTS/GRID-0004-3 (20/03/2009)
- DTS/GRID-0004-4 (24/03/2010)
VirtuaLinux
VirtuaLinux is a Linux meta-distribution that allows the creation, deployment and administration of virtualized clusters with no single point of failure. VirtuaLinux architecture supports disk-less configurations and provides an efficient, iSCSI based abstraction of the SAN. Clusters running VirtuaLinux exhibits no master node to boost resilience and flexibility. Thanks to its storage virtualisation layer, VIrtuaLinux was able to deploy hundreds of VMs in a few seconds. Actually, VirtuaLinux realises a cloud (but the cloud word with the current meaning did not exist in 2006).
Muskel
Muskel is a parallel programming library providing users with structured parallel constructs (skeletons) that can be used to implement efficient parallel applications. Muskel applications run on networks/clusters of workstations equipped with Java (1.5 or greater). The skeletons are implemented exploiting macro data flow technology. Muskel extends Lithium with many interesting features, in particular with adaptive and autonomic features.
Ad-HOC
AD-HOC (Adaptive Distributed Herd of Object Caches), is a fast and robust distributed object repository. It provides applications with a distributed storage manager that virtualise PC’s memories into a unique common distributed storage space. Ad-HOC can effectively be used to implement DSMs as well as distributed cache subsystems, a high-performance distributed shared memory server for cluster and grid. ADHOC is a basic block enabling the development of shared memory run-time supports and applications for dynamic and unreliable executing environments (C++, GPL). The libraries and applications developed on top of ADHOC include:
- a parallel file system exhibiting the same API and better performance of the PVFS;
- a distributed cache that can be plugged in the Apache webserver with no modifications of Apache code, substantially improving web server farm performance;
- a Distributed Shared Memory (DSM) for ASSIST.
ASSIST
ASSIST (A Software development System based on Integrated Skeleton Technology) is a parallel programming environment based on skeleton and coordination language technology aimed at the development of distributed high-performance applications. ASSIST applications should be compiled in binary packages that can be deployed and run on grids, including those exhibiting heterogeneous platforms. Deployment and run are provided through standard middleware services (e.g. Globus) enriched with the ASSIST run-time support. ASSIST applications are described by means of a coordination language, which can express arbitrary graphs of modules, interconnected by typed streams of data. For more information see ASSIST papers.
Lithium
Lithium is a Java-based parallel programming library providing users with structured parallel constructs (patterns/skeletons) that can be used to implement efficient parallel applications on clusters. The skeletons (including pipe, farm, map, reduce, loop) are implemented exploiting macro data flow technology. Lithium skeletons admit a formal specification of both functional and extra-functional behaviour.
Eskimo
Eskimo (Easy SKeleton Interface – Memory Oriented), which was part of my PhD dissertation, is a first (maybe a bit naive) tentative to bring skeletal/pattern-based programming on the shared memory model. To my knowledge, there were no previous experiments since skeletal programming was exclusively living in the message passing arena. From a certain viewpoint, it can be considered an ancestor of Fastflow (and other libraries in this class, such as Intel TBB).
Meta
META is a toolkit for the source-to-source optimisation of pattern-based/skeletal parallel programs (OCaml, GPL). It includes a quite efficient subtree-matching implementation.
SkIE
SkIE (Skeleton-based Integrated Environment) is a skeleton-based parallel programming environment. SkIE was an engineered version of P3L developed within Quadrics Supercomputing World (QSW) and Alenia Aerospace. Within QSW, I have designed and developed part of the compiler back-end.