Software & tools | Parallel Computing
RISC-V tools
We actively contribute to the RISC-V ecosystem via different software ports. As of today, these include:
- Fastflow. Fastflow is a C++ framework for high-level pattern-based parallel programming performance (more info here). The RISC-V port is part of the official repository.
- Pytorch. PyTorch is one of the most popular Python/C++ frameworks for training and using DNN models. The RISC-V port is available here.
- OpenFL for RISC-V. We managed to port the official Intel® OpenFL federated learning framework to the RISC-V platform. The Python packages are available to be installed via pip from this repository.
To properly add our proprietary repository to your pip configuration, just runpip config set global.index-url https://gitlab.di.unito.it/api/v4/projects/1057/packages/pypi/simple
Then, to install OpenFL built for RISC-V, just runpip install openfl-riscv
As side results, on this repository are also available the following RISC-V compatible Python packages: ninja (ninja-riscv), meson-python (meson-python-riscv), scipy (scipy-riscv), scikit-learn (scikit-learn-riscv).
Publications
2023
Gianluca Mittone, Walter Riviera, Iacopo Colonnelli, Robert Birke, Marco Aldinucci
Model-Agnostic Federated Learning Proceedings Article
In: Euro-Par 2023: Parallel Processing, pp. 383–396, Springer, Limassol, Cyprus, 2023.
Abstract | Links | BibTeX | Tags: ai, confidential, eupilot, icsc, riscv
@inproceedings{23:mittone:mafl,
title = {Model-Agnostic Federated Learning},
author = {Gianluca Mittone and Walter Riviera and Iacopo Colonnelli and Robert Birke and Marco Aldinucci},
url = {https://doi.org/10.1007/978-3-031-39698-4_26},
doi = {10.1007/978-3-031-39698-4_26},
year = {2023},
date = {2023-08-01},
booktitle = {Euro-Par 2023: Parallel Processing},
volume = {14100},
pages = {383–396},
publisher = {Springer},
address = {Limassol, Cyprus},
institution = {Computer Science Department, University of Torino},
abstract = {Since its debut in 2016, Federated Learning (FL) has been tied to the inner workings of Deep Neural Networks (DNNs). On the one hand, this allowed its development and widespread use as DNNs proliferated. On the other hand, it neglected all those scenarios in which using DNNs is not possible or advantageous. The fact that most current FL frameworks only allow training DNNs reinforces this problem. To address the lack of FL solutions for non-DNN-based use cases, we propose MAFL (Model-Agnostic Federated Learning). MAFL marries a model-agnostic FL algorithm, AdaBoost.F, with an open industry-grade FL framework: Intel OpenFL. MAFL is the first FL system not tied to any specific type of machine learning model, allowing exploration of FL scenarios beyond DNNs and trees. We test MAFL from multiple points of view, assessing its correctness, flexibility and scaling properties up to 64 nodes. We optimised the base software achieving a 5.5x speedup on a standard FL scenario. MAFL is compatible with x86-64, ARM-v8, Power and RISC-V.},
keywords = {ai, confidential, eupilot, icsc, riscv},
pubstate = {published},
tppubtype = {inproceedings}
}
Since its debut in 2016, Federated Learning (FL) has been tied to the inner workings of Deep Neural Networks (DNNs). On the one hand, this allowed its development and widespread use as DNNs proliferated. On the other hand, it neglected all those scenarios in which using DNNs is not possible or advantageous. The fact that most current FL frameworks only allow training DNNs reinforces this problem. To address the lack of FL solutions for non-DNN-based use cases, we propose MAFL (Model-Agnostic Federated Learning). MAFL marries a model-agnostic FL algorithm, AdaBoost.F, with an open industry-grade FL framework: Intel OpenFL. MAFL is the first FL system not tied to any specific type of machine learning model, allowing exploration of FL scenarios beyond DNNs and trees. We test MAFL from multiple points of view, assessing its correctness, flexibility and scaling properties up to 64 nodes. We optimised the base software achieving a 5.5x speedup on a standard FL scenario. MAFL is compatible with x86-64, ARM-v8, Power and RISC-V.
Iacopo Colonnelli, Robert Birke, Marco Aldinucci
Experimenting with PyTorch on RISC-V Proceedings Article
In: RISC-V Summit Europe 2023, Barcelona, Spain, 2023, (Poster).
Abstract | Links | BibTeX | Tags: eupilot, icsc, riscv
@inproceedings{23:risc-v-summit,
title = {Experimenting with PyTorch on RISC-V},
author = {Iacopo Colonnelli and Robert Birke and Marco Aldinucci},
url = {https://iris.unito.it/retrieve/429bf344-9090-42c3-809c-1b8ac320a930/2023-06-08-Iacopo-COLONNELLI-abstract.pdf},
year = {2023},
date = {2023-06-01},
booktitle = {RISC-V Summit Europe 2023},
address = {Barcelona, Spain},
abstract = {RISC-V is an emerging instruction set architecture. Its modular and extensible open-source royalty-free design is increasingly attracting interest from both research and industry. Nowadays, different RISC-V-based boards can be bought off the shelf. However, software availability is equivalently vital in guaranteeing the RISC-V ecosystem's success. Here we contribute with the first publicly available port of PyTorch. PyTorch is one of the most popular Deep Learning libraries available today. As such, it is a crucial enabler in running state-of-the-art AI applications on RISC-V-based systems and a first step towards a fully democratic end-to-end codesign process.},
note = {Poster},
keywords = {eupilot, icsc, riscv},
pubstate = {published},
tppubtype = {inproceedings}
}
RISC-V is an emerging instruction set architecture. Its modular and extensible open-source royalty-free design is increasingly attracting interest from both research and industry. Nowadays, different RISC-V-based boards can be bought off the shelf. However, software availability is equivalently vital in guaranteeing the RISC-V ecosystem's success. Here we contribute with the first publicly available port of PyTorch. PyTorch is one of the most popular Deep Learning libraries available today. As such, it is a crucial enabler in running state-of-the-art AI applications on RISC-V-based systems and a first step towards a fully democratic end-to-end codesign process.
Gianluca Mittone, Nicolò Tonci, Robert Birke, Iacopo Colonnelli, Doriana Medić, Andrea Bartolini, Roberto Esposito, Emanuele Parisi, Francesco Beneventi, Mirko Polato, Massimo Torquati, Luca Benini, Marco Aldinucci
Experimenting with Emerging RISC-V Systems for Decentralised Machine Learning Proceedings Article
In: 20th ACM International Conference on Computing Frontiers (CF '23), ACM, Bologna, Italy, 2023, ISBN: 979-8-4007-0140-5/23/05, (https://arxiv.org/abs/2302.07946).
Abstract | Links | BibTeX | Tags: ai, confidential, eupilot, HPC, icsc, riscv
@inproceedings{23:mittone:fl-riscv,
title = {Experimenting with Emerging RISC-V Systems for Decentralised Machine Learning},
author = {Gianluca Mittone and Nicolò Tonci and Robert Birke and Iacopo Colonnelli and Doriana Medić and Andrea Bartolini and Roberto Esposito and Emanuele Parisi and Francesco Beneventi and Mirko Polato and Massimo Torquati and Luca Benini and Marco Aldinucci},
url = {https://dl.acm.org/doi/pdf/10.1145/3587135.3592211},
doi = {10.1145/3587135.3592211},
isbn = {979-8-4007-0140-5/23/05},
year = {2023},
date = {2023-05-01},
booktitle = {20th ACM International Conference on Computing Frontiers (CF '23)},
publisher = {ACM},
address = {Bologna, Italy},
institution = {Computer Science Department, University of Torino},
abstract = {Decentralised Machine Learning (DML) enables collaborative machine learning without centralised input data. Federated Learning (FL) and Edge Inference are examples of DML. While tools for DML (especially FL) are starting to flourish, many are not flexible and portable enough to experiment with novel systems (e.g., RISC-V), non-fully connected topologies, and asynchronous collaboration schemes. We overcome these limitations via a domain-specific language allowing to map DML schemes to an underlying middleware, i.e. the FastFlow parallel programming library. We experiment with it by generating different working DML schemes on two emerging architectures (ARM-v8, RISC-V) and the x86-64 platform. We characterise the performance and energy efficiency of the presented schemes and systems. As a byproduct, we introduce a RISC-V porting of the PyTorch framework, the first publicly available to our knowledge.},
note = {https://arxiv.org/abs/2302.07946},
keywords = {ai, confidential, eupilot, HPC, icsc, riscv},
pubstate = {published},
tppubtype = {inproceedings}
}
Decentralised Machine Learning (DML) enables collaborative machine learning without centralised input data. Federated Learning (FL) and Edge Inference are examples of DML. While tools for DML (especially FL) are starting to flourish, many are not flexible and portable enough to experiment with novel systems (e.g., RISC-V), non-fully connected topologies, and asynchronous collaboration schemes. We overcome these limitations via a domain-specific language allowing to map DML schemes to an underlying middleware, i.e. the FastFlow parallel programming library. We experiment with it by generating different working DML schemes on two emerging architectures (ARM-v8, RISC-V) and the x86-64 platform. We characterise the performance and energy efficiency of the presented schemes and systems. As a byproduct, we introduce a RISC-V porting of the PyTorch framework, the first publicly available to our knowledge.
William Fornaciari, Federico Reghenzani, Federico Terraneo, Davide Baroffio, Cecilia Metra, Martin Omana, Josie E. Rodriguez Condia, Matteo Sonza Reorda, Robert Birke, Iacopo Colonnelli, Gianluca Mittone, Marco Aldinucci, Gabriele Mencagli, Francesco Iannone, Filippo Palombi, Giuseppe Zummo, Daniele Cesarini, Federico Tesser
RISC-V-based Platforms for HPC: Analyzing Non-functional Properties for Future HPC and Big-Data Clusters Proceedings Article
In: Embedded Computer Systems: Architectures, Modeling, and Simulation - 23rd International Conference, SAMOS 2023, Samos, Greece, 2023, (icsc).
Abstract | Links | BibTeX | Tags: icsc, riscv
@inproceedings{23:SAMOS,
title = {RISC-V-based Platforms for HPC: Analyzing Non-functional Properties for Future HPC and Big-Data Clusters},
author = {William Fornaciari and Federico Reghenzani and Federico Terraneo and Davide Baroffio and Cecilia Metra and Martin Omana and Josie E. Rodriguez Condia and Matteo Sonza Reorda and Robert Birke and Iacopo Colonnelli and Gianluca Mittone and Marco Aldinucci and Gabriele Mencagli and Francesco Iannone and Filippo Palombi and Giuseppe Zummo and Daniele Cesarini and Federico Tesser},
url = {https://iris.unito.it/retrieve/b627eab0-3aa1-4fd7-8685-f47c62c792b3/SAMOS_2023_CN_HPC_FL1.pdf},
doi = {10.1007/978-3-031-46077-7_26},
year = {2023},
date = {2023-01-01},
booktitle = {Embedded Computer Systems: Architectures, Modeling, and Simulation - 23rd International Conference, SAMOS 2023},
address = {Samos, Greece},
abstract = {High-PerformanceComputing(HPC)haveevolvedtobeused to perform simulations of systems where physical experimentation is pro- hibitively impractical, expensive, or dangerous. This paper provides a general overview and showcases the analysis of non-functional properties in RISC-V-based platforms for HPCs. In particular, our analyses target the evaluation of power and energy control, thermal management, and reliability assessment of promising systems, structures, and technologies devised for current and future generation of HPC machines. The main set of design methodologies and technologies developed within the activ- ities of the Future and HPC & Big Data spoke of the National Centre of HPC, Big Data and Quantum Computing project are described along with the description of the testbed for experimenting two-phase cooling approaches.},
note = {icsc},
keywords = {icsc, riscv},
pubstate = {published},
tppubtype = {inproceedings}
}
High-PerformanceComputing(HPC)haveevolvedtobeused to perform simulations of systems where physical experimentation is pro- hibitively impractical, expensive, or dangerous. This paper provides a general overview and showcases the analysis of non-functional properties in RISC-V-based platforms for HPCs. In particular, our analyses target the evaluation of power and energy control, thermal management, and reliability assessment of promising systems, structures, and technologies devised for current and future generation of HPC machines. The main set of design methodologies and technologies developed within the activ- ities of the Future and HPC & Big Data spoke of the National Centre of HPC, Big Data and Quantum Computing project are described along with the description of the testbed for experimenting two-phase cooling approaches.
Talks
Sorry, no publications matched your criteria.
FastFederatedLearning
Fast Federated Learning (FFL) is a C/C++-based Federated Learning framework built on top of the parallel programming FastFlow framework. It exploits the Cereal library to efficiently serialise the updates sent over the network and the libtorch library to fully bypass the need for Python code. The first release of this software comprises three examples based on three different communication topologies: master-worker, peer-to-peer, and tree-based.
FastFederatedLearning is freely available on GitHub under the LGPLv3 license. It has been successfully tested on x86_64, ARM, and RISC-V platforms. FFL has scripts for automatically installing the framework and reproducing all the experiments reported in the original paper. More information about software usage can be found on the official repository.
G. Mittone, N. Tonci, R. Birke, I. Colonnelli, D. Medić, A. Bartolini, R. Esposito, E. Parisi, F. Beneventi, M. Polato, M. Torquati, L. Benini, and M. Aldinucci, “Experimenting with Emerging RISC-V Systems for Decentralised Machine Learning“, 20th ACM International Conference on Computing Frontiers, 2023. DOI: 10.1145/3587135.3592211
Publications
Sorry, no publications matched your criteria.
Talks
Sorry, no publications matched your criteria.
OpenFL-extended
OpenFL-extended is an extended version of the official Intel® OpenFL federated learning (FL) framework. OpenFL-extended fully supports the standard FL workflow already provided by OpenFL, but in addition, it provides support for both federated bagging and federated boosting approaches. Federated bagging is implemented through simple bagging of models trained by different parties from the aggregator, while federated boosting is obtained employing the AdaBoost.F algorithm developed at the University of Turin[1]. Through these approaches, OpenFL extended is fully model-agnostic, which means that it can be used to build federations out of any Machine Learning model, not only Deep Neural Networks.
OpenFL extended is freely available on GitHub under the LGPLv3 license. It is fully Python-based and comes with a wide range of ready-made examples. It has been tested on x86_64, ARM and RISC-V architectures. More information about software usage can be found on the official repository.
This software’s publication is currently under review, but an open-access version of the paper is available on arXiv.
G. Mittone, W. Riviera, I. Colonnelli, R. Birke, M. Aldinucci, “Model-Agnostic Federated Learning“, arXive, 2023. DOI: 10.48550/arXiv.2303.04906
[1] M. Polato, R. Esposito, and M. Aldinucci. “Boosting the federation: Cross-silo federated learning without gradient descent.” 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022.
Publications
Sorry, no publications matched your criteria.
Talks
Sorry, no publications matched your criteria.
Jupyter Workflow
Jupyter Workflow is an extension of the IPython kernel designed to support distributed literate workflows. The Jupyter Workflow kernel enables Jupyter Notebooks to describe complex workflows and to execute them in a distributed fashion on hybrid cloud/HPC infrastructures. In particular, code cells are regarded as the nodes of a distributed workflow graph, whereas cell metadata are used to express data and control dependencies, parallel execution patterns (e.g. Scatter/Gather), and target execution infrastructures.
Jupyter Workflow code is available on GitHub under the LGPLv3 license, and the related Python package is downloadable from PyPI. More details about the tool and its applications can be found on the Jupyter Workkflow website.
I. Colonnelli, M. Aldinucci, B. Cantalupo, L. Padovani, S. Rabellino, C. Spampinato, R. Morelli, R. Di Carlo, N. Magini and C. Cavazzoni, “Distributed workflows with Jupyter”, Future Generation Computer Systems, vol. 128, pp. 282-298, 2022. doi: 10.1016/j.future.2021.10.007.
Publications
2023
Marco Aldinucci, Elena Maria Baralis, Valeria Cardellini, Iacopo Colonnelli, Marco Danelutto, Sergio Decherchi, Giuseppe Di Modica, Luca Ferrucci, Marco Gribaudo, Francesco Iannone, Marco Lapegna, Doriana Medic, Giuseppa Muscianisi, Francesca Righetti, Eva Sciacca, Nicola Tonellotto, Mauro Tortonesi, Paolo Trunfio, Tullio Vardanega
A Systematic Mapping Study of Italian Research on Workflows Proceedings Article
In: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023, pp. 2065–2076, ACM, Denver, CO, USA, 2023.
Abstract | Links | BibTeX | Tags: icsc, jupyter-workflow, streamflow
@inproceedings{WORKS2023,
title = {A Systematic Mapping Study of Italian Research on Workflows},
author = {Marco Aldinucci and Elena Maria Baralis and Valeria Cardellini and Iacopo Colonnelli and Marco Danelutto and Sergio Decherchi and Giuseppe Di Modica and Luca Ferrucci and Marco Gribaudo and Francesco Iannone and Marco Lapegna and Doriana Medic and Giuseppa Muscianisi and Francesca Righetti and Eva Sciacca and Nicola Tonellotto and Mauro Tortonesi and Paolo Trunfio and Tullio Vardanega},
url = {https://doi.org/10.1145/3624062.3624285},
doi = {10.1145/3624062.3624285},
year = {2023},
date = {2023-11-01},
booktitle = {Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023},
pages = {2065–2076},
publisher = {ACM},
address = {Denver, CO, USA},
abstract = {An entire ecosystem of methodologies and tools revolves around scientific workflow management. They cover crucial non-functional requirements that standard workflow models fail to target, such as interactive execution, energy efficiency, performance portability, Big Data management, and intelligent orchestration in the Computing Continuum. Characterizing and monitoring this ecosystem is crucial to develop an informed view of current and future research directions. This work conducts a systematic mapping study of the Italian workflow research community, collecting and analyzing 25 tools and 10 applications from several scientific domains in the context of the ``National Research Centre for HPC, Big Data, and Quantum Computing'' (ICSC). The study aims to outline the main current research directions and determine how they address the critical needs of modern scientific applications. The findings highlight a variegated research ecosystem of tools, with a prominent interest in advanced workflow orchestration and still immature but promising efforts toward energy efficiency.},
keywords = {icsc, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {inproceedings}
}
An entire ecosystem of methodologies and tools revolves around scientific workflow management. They cover crucial non-functional requirements that standard workflow models fail to target, such as interactive execution, energy efficiency, performance portability, Big Data management, and intelligent orchestration in the Computing Continuum. Characterizing and monitoring this ecosystem is crucial to develop an informed view of current and future research directions. This work conducts a systematic mapping study of the Italian workflow research community, collecting and analyzing 25 tools and 10 applications from several scientific domains in the context of the ``National Research Centre for HPC, Big Data, and Quantum Computing'' (ICSC). The study aims to outline the main current research directions and determine how they address the critical needs of modern scientific applications. The findings highlight a variegated research ecosystem of tools, with a prominent interest in advanced workflow orchestration and still immature but promising efforts toward energy efficiency.
Iacopo Colonnelli
Workflow Models for Heterogeneous Distributed Systems Proceedings Article
In: Bena, Nicola, Martino, Beniamino Di, Maratea, Antonio, Sperduti, Alessandro, Nardo, Emanuel Di, Ciaramella, Angelo, Montella, Raffaele, Ardagna, Claudio A. (Ed.): Proceedings of the 2nd Italian Conference on Big Data and Data Science (ITADATA 2023), Naples, Italy, September 11-13, 2023, CEUR-WS.org, 2023.
Abstract | Links | BibTeX | Tags: across, eupex, icsc, jupyter-workflow, streamflow
@inproceedings{23:colonnelli:itadata,
title = {Workflow Models for Heterogeneous Distributed Systems},
author = {Iacopo Colonnelli},
editor = {Nicola Bena and Beniamino Di Martino and Antonio Maratea and Alessandro Sperduti and Emanuel Di Nardo and Angelo Ciaramella and Raffaele Montella and Claudio A. Ardagna},
url = {https://ceur-ws.org/Vol-3606/invited77.pdf},
year = {2023},
date = {2023-01-01},
booktitle = {Proceedings of the 2nd Italian Conference on Big Data and Data Science (ITADATA 2023), Naples, Italy, September 11-13, 2023},
volume = {3606},
publisher = {CEUR-WS.org},
series = {CEUR Workshop Proceedings},
abstract = {This article introduces a novel hybrid workflow abstraction that injects topology awareness directly into the definition of a distributed workflow model. In particular, the article briefly discusses the advantages brought by this approach to the design and orchestration of large-scale data-oriented workflows, the current level of support from state-of-the-art workflow systems, and some future research directions.},
keywords = {across, eupex, icsc, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {inproceedings}
}
This article introduces a novel hybrid workflow abstraction that injects topology awareness directly into the definition of a distributed workflow model. In particular, the article briefly discusses the advantages brought by this approach to the design and orchestration of large-scale data-oriented workflows, the current level of support from state-of-the-art workflow systems, and some future research directions.
2022
Iacopo Colonnelli, Marco Aldinucci, Barbara Cantalupo, Luca Padovani, Sergio Rabellino, Concetto Spampinato, Roberto Morelli, Rosario Di Carlo, Nicolò Magini, Carlo Cavazzoni
Distributed workflows with Jupyter Journal Article
In: Future Generation Computer Systems, vol. 128, pp. 282–298, 2022, ISSN: 0167-739X.
Abstract | Links | BibTeX | Tags: across, deephealth, jupyter-workflow, streamflow
@article{21:FGCS:jupyflow,
title = {Distributed workflows with Jupyter},
author = {Iacopo Colonnelli and Marco Aldinucci and Barbara Cantalupo and Luca Padovani and Sergio Rabellino and Concetto Spampinato and Roberto Morelli and Rosario Di Carlo and Nicolò Magini and Carlo Cavazzoni},
url = {https://www.sciencedirect.com/science/article/pii/S0167739X21003976},
doi = {10.1016/j.future.2021.10.007},
issn = {0167-739X},
year = {2022},
date = {2022-01-01},
journal = {Future Generation Computer Systems},
volume = {128},
pages = {282–298},
abstract = {The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code's business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks' potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.},
keywords = {across, deephealth, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {article}
}
The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code's business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks' potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.
Talks
2024
Iacopo Colonnelli
Dynamic hybrid workflows for Deep Learning on HPC infrastructure Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: icsc, jupyter-workflow, streamflow
@misc{24:icolonne:ictp,
title = {Dynamic hybrid workflows for Deep Learning on HPC infrastructure},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/EaFHJEKNbW5oXeq},
year = {2024},
date = {2024-05-01},
address = {Trieste, Italy},
abstract = {Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher’s laptop to an entire HPC facility.},
keywords = {icsc, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher’s laptop to an entire HPC facility.
2023
Iacopo Colonnelli
Workflow models for heterogeneous distributed systems Miscellaneous
2nd Italian Conference on Big Data and Data Science (ITADATA 2023), 2023, (Best PhD Thesis Award).
Links | BibTeX | Tags: jupyter-workflow, streamflow
@misc{23:ITADATABestPhDThesis,
title = {Workflow models for heterogeneous distributed systems},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/6RqcaJ4djqFNDC8},
year = {2023},
date = {2023-09-01},
address = {Napoli, Italy},
howpublished = {2nd Italian Conference on Big Data and Data Science (ITADATA 2023)},
note = {Best PhD Thesis Award},
keywords = {jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
UNITO tools presentation Miscellaneous
CN HPC Flagship 3 Working Day, 2023.
Links | BibTeX | Tags: jupyter-workflow, streamflow
@misc{23:FL3WorkingDay,
title = {UNITO tools presentation},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/fgHbnLDQSFtcwLd},
year = {2023},
date = {2023-05-01},
address = {Bologna, Italy},
howpublished = {CN HPC Flagship 3 Working Day},
keywords = {jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
2022
Iacopo Colonnelli, Marco Aldinucci
Hybrid Workflows For Large-Scale Scientific Applications Miscellaneous
6th EAGE High Performance Computing Workshop, 2022.
Abstract | Links | BibTeX | Tags: across, eupex, jupyter-workflow, textarossa
@misc{22:eage,
title = {Hybrid Workflows For Large-Scale Scientific Applications},
author = {Iacopo Colonnelli and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/GScPS5LCPdt6Yoo},
year = {2022},
date = {2022-09-01},
address = {Milano, Italy},
abstract = {Large-scale scientific applications are facing an irreversible transition from monolithic, high-performance oriented codes to modular and polyglot deployments of specialised (micro-)services. The reasons behind this transition are many: coupling of standard solvers with Deep Learning techniques, offloading of data analysis and visualisation to Cloud, and the advent of specialised hardware accelerators. Topology-aware Workflow Management Systems (WMSs) play a crucial role. In particular, topology-awareness allows an explicit mapping of workflow steps onto heterogeneous locations, allowing automated executions on top of hybrid architectures (e.g., cloud+HPC or classical+quantum). Plus, topology-aware WMSs can offer non-functional requirements OOTB, e.g. components’ life-cycle orchestration, secure and efficient data transfers, fault tolerance, and cross-cluster execution of urgent workloads. Augmenting interactive Jupyter Notebooks with distributed workflow capabilities allows domain experts to prototype and scale applications using the same technological stack, while relying on a feature-rich and user-friendly web interface. This abstract will showcase how these general methodologies can be applied to a typical geoscience simulation pipeline based on the Full Wavefront Inversion (FWI) technique. In particular, a prototypical Jupyter Notebook will be executed interactively on Cloud. Preliminary data analyses and post-processing will be executed locally, while the computationally demanding optimisation loop will be scheduled on a remote HPC cluster.},
howpublished = {6th EAGE High Performance Computing Workshop},
keywords = {across, eupex, jupyter-workflow, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Large-scale scientific applications are facing an irreversible transition from monolithic, high-performance oriented codes to modular and polyglot deployments of specialised (micro-)services. The reasons behind this transition are many: coupling of standard solvers with Deep Learning techniques, offloading of data analysis and visualisation to Cloud, and the advent of specialised hardware accelerators. Topology-aware Workflow Management Systems (WMSs) play a crucial role. In particular, topology-awareness allows an explicit mapping of workflow steps onto heterogeneous locations, allowing automated executions on top of hybrid architectures (e.g., cloud+HPC or classical+quantum). Plus, topology-aware WMSs can offer non-functional requirements OOTB, e.g. components’ life-cycle orchestration, secure and efficient data transfers, fault tolerance, and cross-cluster execution of urgent workloads. Augmenting interactive Jupyter Notebooks with distributed workflow capabilities allows domain experts to prototype and scale applications using the same technological stack, while relying on a feature-rich and user-friendly web interface. This abstract will showcase how these general methodologies can be applied to a typical geoscience simulation pipeline based on the Full Wavefront Inversion (FWI) technique. In particular, a prototypical Jupyter Notebook will be executed interactively on Cloud. Preliminary data analyses and post-processing will be executed locally, while the computationally demanding optimisation loop will be scheduled on a remote HPC cluster.
Iacopo Colonnelli, Dario Tranchitella
Dossier: multi-tenant distributed Jupyter Notebooks Miscellaneous
DoK Talks 141, 2022, (Invited talk).
Abstract | Links | BibTeX | Tags: across, deephealth, hpc4ai, jupyter-workflow
@misc{22:data-on-kubernetes,
title = {Dossier: multi-tenant distributed Jupyter Notebooks},
author = {Iacopo Colonnelli and Dario Tranchitella},
url = {https://datacloud.di.unito.it/index.php/s/RNqTGmTqWS66qHT},
year = {2022},
date = {2022-07-01},
address = {Virtual event},
abstract = {When providing data analysis as a service, one must tackle several problems. Data privacy and protection by design are crucial when working on sensitive data. Performance and scalability are fundamental for compute-intensive workloads, e.g. training Deep Neural Networks. User-friendly interfaces and fast prototyping tools are essential to allow domain experts to experiment with new techniques. Portability and reproducibility are necessary to assess the actual value of results. Kubernetes is the best platform to provide reliable, elastic, and maintainable services. However, Kubernetes alone is not enough to achieve large-scale multi-tenant reproducible data analysis. OOTB support for multi-tenancy is too rough, with only two levels of segregation (i.e. the single namespace or the entire cluster). Offloading computation to off-cluster resources is non-trivial and requires the user's manual configuration. Also, Jupyter Notebooks per se cannot provide much scalability (they execute locally and sequentially) and reproducibility (users can run cells in any order and any number of times). The Dossier platform allows system administrators to manage multi-tenant distributed Jupyter Notebooks at the cluster level in the Kubernetes way, i.e. through CRDs. Namespaces are aggregated in Tenants, and all security and accountability aspects are managed at that level. Each Notebook spawns into a user-dedicated namespace, subject to all Tenant-level constraints. Users can rely on provisioned resources, either in-cluster worker nodes or external resources like HPC facilities. Plus, they can plug their computing nodes in a BYOD fashion. Notebooks are interpreted as distributed workflows, where each cell is a task that one can offload to a different location in charge of its execution.},
howpublished = {DoK Talks 141},
note = {Invited talk},
keywords = {across, deephealth, hpc4ai, jupyter-workflow},
pubstate = {published},
tppubtype = {misc}
}
When providing data analysis as a service, one must tackle several problems. Data privacy and protection by design are crucial when working on sensitive data. Performance and scalability are fundamental for compute-intensive workloads, e.g. training Deep Neural Networks. User-friendly interfaces and fast prototyping tools are essential to allow domain experts to experiment with new techniques. Portability and reproducibility are necessary to assess the actual value of results. Kubernetes is the best platform to provide reliable, elastic, and maintainable services. However, Kubernetes alone is not enough to achieve large-scale multi-tenant reproducible data analysis. OOTB support for multi-tenancy is too rough, with only two levels of segregation (i.e. the single namespace or the entire cluster). Offloading computation to off-cluster resources is non-trivial and requires the user's manual configuration. Also, Jupyter Notebooks per se cannot provide much scalability (they execute locally and sequentially) and reproducibility (users can run cells in any order and any number of times). The Dossier platform allows system administrators to manage multi-tenant distributed Jupyter Notebooks at the cluster level in the Kubernetes way, i.e. through CRDs. Namespaces are aggregated in Tenants, and all security and accountability aspects are managed at that level. Each Notebook spawns into a user-dedicated namespace, subject to all Tenant-level constraints. Users can rely on provisioned resources, either in-cluster worker nodes or external resources like HPC facilities. Plus, they can plug their computing nodes in a BYOD fashion. Notebooks are interpreted as distributed workflows, where each cell is a task that one can offload to a different location in charge of its execution.
Iacopo Colonnelli, Dario Tranchitella
OpenDeepHealth: Crafting a Deep Learning Platform as a Service with Kubernetes Miscellaneous
J on The Beach 2022, 2022.
Links | BibTeX | Tags: across, deephealth, hpc4ai, jupyter-workflow, streamflow
@misc{22:jotb22,
title = {OpenDeepHealth: Crafting a Deep Learning Platform as a Service with Kubernetes},
author = {Iacopo Colonnelli and Dario Tranchitella},
url = {https://datacloud.di.unito.it/index.php/s/n6J7STNnwdyqtET},
year = {2022},
date = {2022-04-01},
address = {Malaga, Spain},
howpublished = {J on The Beach 2022},
keywords = {across, deephealth, hpc4ai, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
Distributed workflows with Jupyter Miscellaneous
J on The Beach 2022, 2022, (Workshop).
Links | BibTeX | Tags: across, deephealth, jupyter-workflow, streamflow
@misc{22:jotb22-workshop,
title = {Distributed workflows with Jupyter},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/om89q55S6ePf2Ji},
year = {2022},
date = {2022-04-01},
address = {Malaga, Spain},
howpublished = {J on The Beach 2022},
note = {Workshop},
keywords = {across, deephealth, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
2020
Iacopo Colonnelli, Sergio Rabellino
JupyterFlow: Jupyter Notebooks su larga scala Miscellaneous
Workshop GARR 2020, 2020.
Abstract | Links | BibTeX | Tags: deephealth, hpc4ai, jupyter-workflow
@misc{20:GarrWorkshop,
title = {JupyterFlow: Jupyter Notebooks su larga scala},
author = {Iacopo Colonnelli and Sergio Rabellino},
url = {https://datacloud.di.unito.it/index.php/s/ASPEmyXAj5QscgC},
year = {2020},
date = {2020-11-01},
address = {Virtual event},
abstract = {I Jupyter Notebook sono largamente utilizzati sia in ambito industriale che accademico come strumento di didattica, prototipazione e analisi esplorative. Purtroppo il sistema runtime standard di Jupyter non è abbastanza potente per sostenere un carichi di lavoro reali e spesso l'unica soluzione è quella di riscrivere il codice da zero in una tecnologia con supporto HPC. Intrgrando lo stack Jupyter con StreamFlow (https://streamflow.di.unito.it/) è possibile creare i Notebook tramite un'interfaccia web su cloud ed eseguirli in maniera trasparente in remoto su una VM con GPU o su nodi HPC.},
howpublished = {Workshop GARR 2020},
keywords = {deephealth, hpc4ai, jupyter-workflow},
pubstate = {published},
tppubtype = {misc}
}
I Jupyter Notebook sono largamente utilizzati sia in ambito industriale che accademico come strumento di didattica, prototipazione e analisi esplorative. Purtroppo il sistema runtime standard di Jupyter non è abbastanza potente per sostenere un carichi di lavoro reali e spesso l'unica soluzione è quella di riscrivere il codice da zero in una tecnologia con supporto HPC. Intrgrando lo stack Jupyter con StreamFlow (https://streamflow.di.unito.it/) è possibile creare i Notebook tramite un'interfaccia web su cloud ed eseguirli in maniera trasparente in remoto su una VM con GPU o su nodi HPC.
StreamFlow
The StreamFlow framework is a container-native Workflow Management System (WMS) written in Python 3 and based on the Common Workflow Language (CWL) Standard.
StreamFlow has been designed around two main principles:
- Allowing the execution of tasks in multi-container environments in order to support the concurrent execution of multiple communicating tasks in a multi-agent ecosystem
- Relaxing the requirement of a single shared data space to allow for hybrid workflow executions on top of multi-cloud or hybrid cloud/HPC infrastructures.
StreamFlow source code is available on GitHub under the LGPLv3 license. Moreover, a Python package is downloadable from PyPI and Docker containers can be found on Docker Hub. More details about the tool and its applications can be found on the StreamFlow website.
StreamFlow has been selected as an exploring technology by the EC Innovation Radar initiative.
I. Colonnelli, B. Cantalupo, I. Merelli and M. Aldinucci, “StreamFlow: cross-breeding cloud with HPC,” in IEEE Transactions on Emerging Topics in Computing, doi: 10.1109/TETC.2020.3019202.
Publications
2024
Simone Leo, Michael R. Crusoe, Laura Rodríguez-Navas, Raül Sirvent, Alexander Kanitz, Paul De Geest, Rudolf Wittner, Luca Pireddu, Daniel Garijo, José M. Fernández, Iacopo Colonnelli, Matej Gallo, Tazro Ohta, Hirotaka Suetake, Salvador Capella-Gutierrez, Renske Wit, Bruno P. Kinoshita, Stian Soiland-Reyes
Recording provenance of workflow runs with RO-Crate Journal Article
In: PLoS ONE, vol. 19, no. 9, pp. 1–35, 2024.
Abstract | Links | BibTeX | Tags: across, eupex, icsc, streamflow
@article{24:pone:wfrunrocrate,
title = {Recording provenance of workflow runs with RO-Crate},
author = {Simone Leo and Michael R. Crusoe and Laura Rodríguez-Navas and Raül Sirvent and Alexander Kanitz and Paul De Geest and Rudolf Wittner and Luca Pireddu and Daniel Garijo and José M. Fernández and Iacopo Colonnelli and Matej Gallo and Tazro Ohta and Hirotaka Suetake and Salvador Capella-Gutierrez and Renske Wit and Bruno P. Kinoshita and Stian Soiland-Reyes},
url = {https://iris.unito.it/retrieve/57752f7b-9f8f-4013-8cef-5b498703d882/journal.pone.0309210.pdf},
doi = {10.1371/journal.pone.0309210},
year = {2024},
date = {2024-09-01},
journal = {PLoS ONE},
volume = {19},
number = {9},
pages = {1–35},
publisher = {Public Library of Science},
abstract = {Recording the provenance of scientific computation results is key to the support of traceability, reproducibility and quality assessment of data products. Several data models have been explored to address this need, providing representations of workflow plans and their executions as well as means of packaging the resulting information for archiving and sharing. However, existing approaches tend to lack interoperable adoption across workflow management systems. In this work we present Workflow Run RO-Crate, an extension of RO-Crate (Research Object Crate) and Schema.org to capture the provenance of the execution of computational workflows at different levels of granularity and bundle together all their associated objects (inputs, outputs, code, etc.). The model is supported by a diverse, open community that runs regular meetings, discussing development, maintenance and adoption aspects. Workflow Run RO-Crate is already implemented by several workflow management systems, allowing interoperable comparisons between workflow runs from heterogeneous systems. We describe the model, its alignment to standards such as W3C PROV, and its implementation in six workflow systems. Finally, we illustrate the application of Workflow Run RO-Crate in two use cases of machine learning in the digital image analysis domain.},
keywords = {across, eupex, icsc, streamflow},
pubstate = {published},
tppubtype = {article}
}
Recording the provenance of scientific computation results is key to the support of traceability, reproducibility and quality assessment of data products. Several data models have been explored to address this need, providing representations of workflow plans and their executions as well as means of packaging the resulting information for archiving and sharing. However, existing approaches tend to lack interoperable adoption across workflow management systems. In this work we present Workflow Run RO-Crate, an extension of RO-Crate (Research Object Crate) and Schema.org to capture the provenance of the execution of computational workflows at different levels of granularity and bundle together all their associated objects (inputs, outputs, code, etc.). The model is supported by a diverse, open community that runs regular meetings, discussing development, maintenance and adoption aspects. Workflow Run RO-Crate is already implemented by several workflow management systems, allowing interoperable comparisons between workflow runs from heterogeneous systems. We describe the model, its alignment to standards such as W3C PROV, and its implementation in six workflow systems. Finally, we illustrate the application of Workflow Run RO-Crate in two use cases of machine learning in the digital image analysis domain.
Alberto Mulone, Doriana Medić, Marco Aldinucci
A Fault Tolerance mechanism for Hybrid Scientific Workflows Proceedings Article
In: 1st workshop about High-Performance e-Science (HiPES), Madrid, Spain, 2024.
Abstract | BibTeX | Tags: eupex, icsc, streamflow
@inproceedings{24:madrid:hipes,
title = {A Fault Tolerance mechanism for Hybrid Scientific Workflows},
author = {Alberto Mulone and Doriana Medić and Marco Aldinucci},
year = {2024},
date = {2024-08-01},
booktitle = {1st workshop about High-Performance e-Science (HiPES)},
address = {Madrid, Spain},
abstract = {In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.},
keywords = {eupex, icsc, streamflow},
pubstate = {published},
tppubtype = {inproceedings}
}
In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.
Iacopo Colonnelli, Robert Birke, Giulio Malenza, Gianluca Mittone, Alberto Mulone, Jeroen Galjaard, Lydia Y. Chen, Sanzio Bassini, Gabriella Scipione, Jan Martinovič, Vit Vondrák, Marco Aldinucci
Cross-Facility Federated Learning Journal Article
In: Procedia Computer Science, vol. 240, pp. 3–12, 2024, ISSN: 1877-0509.
Abstract | Links | BibTeX | Tags: icsc, space, streamflow
@article{24:eurohpc:xffl,
title = {Cross-Facility Federated Learning},
author = {Iacopo Colonnelli and Robert Birke and Giulio Malenza and Gianluca Mittone and Alberto Mulone and Jeroen Galjaard and Lydia Y. Chen and Sanzio Bassini and Gabriella Scipione and Jan Martinovič and Vit Vondrák and Marco Aldinucci},
url = {https://www.sciencedirect.com/science/article/pii/S1877050924016909},
doi = {10.1016/j.procs.2024.07.003},
issn = {1877-0509},
year = {2024},
date = {2024-01-01},
booktitle = {Proceedings of the First EuroHPC user day},
journal = {Procedia Computer Science},
volume = {240},
pages = {3–12},
publisher = {Elsevier},
address = {Bruxelles, Belgium},
abstract = {In a decade, AI frontier research transitioned from the researcher's workstation to thousands of high-end hardware-accelerated compute nodes. This rapid evolution shows no signs of slowing down in the foreseeable future. While top cloud providers may be able to keep pace with this growth rate, obtaining and efficiently exploiting computing resources at that scale is a daunting challenge for universities and SMEs. This work introduces the Cross-Facility Federated Learning (XFFL) framework to bridge this compute divide, extending the opportunity to efficiently exploit multiple independent data centres for extreme-scale deep learning tasks to data scientists and domain experts. XFFL relies on hybrid workflow abstractions to decouple tasks from environment-specific technicalities, reducing complexity and enhancing reusability. In addition, Federated Learning (FL) algorithms eliminate the need to move large amounts of data between different facilities, reducing time-to-solution and preserving data privacy. The XFFL approach is empirically evaluated by training a full LLaMAv2 7B instance on two facilities of the EuroHPC JU, showing how the increased computing power completely compensates for the additional overhead introduced by two data centres.},
keywords = {icsc, space, streamflow},
pubstate = {published},
tppubtype = {article}
}
In a decade, AI frontier research transitioned from the researcher's workstation to thousands of high-end hardware-accelerated compute nodes. This rapid evolution shows no signs of slowing down in the foreseeable future. While top cloud providers may be able to keep pace with this growth rate, obtaining and efficiently exploiting computing resources at that scale is a daunting challenge for universities and SMEs. This work introduces the Cross-Facility Federated Learning (XFFL) framework to bridge this compute divide, extending the opportunity to efficiently exploit multiple independent data centres for extreme-scale deep learning tasks to data scientists and domain experts. XFFL relies on hybrid workflow abstractions to decouple tasks from environment-specific technicalities, reducing complexity and enhancing reusability. In addition, Federated Learning (FL) algorithms eliminate the need to move large amounts of data between different facilities, reducing time-to-solution and preserving data privacy. The XFFL approach is empirically evaluated by training a full LLaMAv2 7B instance on two facilities of the EuroHPC JU, showing how the increased computing power completely compensates for the additional overhead introduced by two data centres.
2023
Marco Aldinucci, Elena Maria Baralis, Valeria Cardellini, Iacopo Colonnelli, Marco Danelutto, Sergio Decherchi, Giuseppe Di Modica, Luca Ferrucci, Marco Gribaudo, Francesco Iannone, Marco Lapegna, Doriana Medic, Giuseppa Muscianisi, Francesca Righetti, Eva Sciacca, Nicola Tonellotto, Mauro Tortonesi, Paolo Trunfio, Tullio Vardanega
A Systematic Mapping Study of Italian Research on Workflows Proceedings Article
In: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023, pp. 2065–2076, ACM, Denver, CO, USA, 2023.
Abstract | Links | BibTeX | Tags: icsc, jupyter-workflow, streamflow
@inproceedings{WORKS2023,
title = {A Systematic Mapping Study of Italian Research on Workflows},
author = {Marco Aldinucci and Elena Maria Baralis and Valeria Cardellini and Iacopo Colonnelli and Marco Danelutto and Sergio Decherchi and Giuseppe Di Modica and Luca Ferrucci and Marco Gribaudo and Francesco Iannone and Marco Lapegna and Doriana Medic and Giuseppa Muscianisi and Francesca Righetti and Eva Sciacca and Nicola Tonellotto and Mauro Tortonesi and Paolo Trunfio and Tullio Vardanega},
url = {https://doi.org/10.1145/3624062.3624285},
doi = {10.1145/3624062.3624285},
year = {2023},
date = {2023-11-01},
booktitle = {Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023},
pages = {2065–2076},
publisher = {ACM},
address = {Denver, CO, USA},
abstract = {An entire ecosystem of methodologies and tools revolves around scientific workflow management. They cover crucial non-functional requirements that standard workflow models fail to target, such as interactive execution, energy efficiency, performance portability, Big Data management, and intelligent orchestration in the Computing Continuum. Characterizing and monitoring this ecosystem is crucial to develop an informed view of current and future research directions. This work conducts a systematic mapping study of the Italian workflow research community, collecting and analyzing 25 tools and 10 applications from several scientific domains in the context of the ``National Research Centre for HPC, Big Data, and Quantum Computing'' (ICSC). The study aims to outline the main current research directions and determine how they address the critical needs of modern scientific applications. The findings highlight a variegated research ecosystem of tools, with a prominent interest in advanced workflow orchestration and still immature but promising efforts toward energy efficiency.},
keywords = {icsc, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {inproceedings}
}
An entire ecosystem of methodologies and tools revolves around scientific workflow management. They cover crucial non-functional requirements that standard workflow models fail to target, such as interactive execution, energy efficiency, performance portability, Big Data management, and intelligent orchestration in the Computing Continuum. Characterizing and monitoring this ecosystem is crucial to develop an informed view of current and future research directions. This work conducts a systematic mapping study of the Italian workflow research community, collecting and analyzing 25 tools and 10 applications from several scientific domains in the context of the ``National Research Centre for HPC, Big Data, and Quantum Computing'' (ICSC). The study aims to outline the main current research directions and determine how they address the critical needs of modern scientific applications. The findings highlight a variegated research ecosystem of tools, with a prominent interest in advanced workflow orchestration and still immature but promising efforts toward energy efficiency.
Iacopo Colonnelli
Workflow Models for Heterogeneous Distributed Systems Proceedings Article
In: Bena, Nicola, Martino, Beniamino Di, Maratea, Antonio, Sperduti, Alessandro, Nardo, Emanuel Di, Ciaramella, Angelo, Montella, Raffaele, Ardagna, Claudio A. (Ed.): Proceedings of the 2nd Italian Conference on Big Data and Data Science (ITADATA 2023), Naples, Italy, September 11-13, 2023, CEUR-WS.org, 2023.
Abstract | Links | BibTeX | Tags: across, eupex, icsc, jupyter-workflow, streamflow
@inproceedings{23:colonnelli:itadata,
title = {Workflow Models for Heterogeneous Distributed Systems},
author = {Iacopo Colonnelli},
editor = {Nicola Bena and Beniamino Di Martino and Antonio Maratea and Alessandro Sperduti and Emanuel Di Nardo and Angelo Ciaramella and Raffaele Montella and Claudio A. Ardagna},
url = {https://ceur-ws.org/Vol-3606/invited77.pdf},
year = {2023},
date = {2023-01-01},
booktitle = {Proceedings of the 2nd Italian Conference on Big Data and Data Science (ITADATA 2023), Naples, Italy, September 11-13, 2023},
volume = {3606},
publisher = {CEUR-WS.org},
series = {CEUR Workshop Proceedings},
abstract = {This article introduces a novel hybrid workflow abstraction that injects topology awareness directly into the definition of a distributed workflow model. In particular, the article briefly discusses the advantages brought by this approach to the design and orchestration of large-scale data-oriented workflows, the current level of support from state-of-the-art workflow systems, and some future research directions.},
keywords = {across, eupex, icsc, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {inproceedings}
}
This article introduces a novel hybrid workflow abstraction that injects topology awareness directly into the definition of a distributed workflow model. In particular, the article briefly discusses the advantages brought by this approach to the design and orchestration of large-scale data-oriented workflows, the current level of support from state-of-the-art workflow systems, and some future research directions.
Alberto Mulone, Sherine Awad, Davide Chiarugi, Marco Aldinucci
Porting the Variant Calling Pipeline for NGS data in cloud-HPC environment Proceedings Article
In: Shahriar, Hossain, Teranishi, Yuuichi, Cuzzocrea, Alfredo, Sharmin, Moushumi, Towey, Dave, Majumder, A. K. M. Jahangir Alam, Kashiwazaki, Hiroki, Yang, Ji-Jiang, Takemoto, Michiharu, Sakib, Nazmus, Banno, Ryohei, Ahamed, Sheikh Iqbal (Ed.): 47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023, pp. 1858–1863, IEEE, Torino, Italy, 2023.
Abstract | Links | BibTeX | Tags: across, icsc, streamflow
@inproceedings{23:mulone:wide:vcp,
title = {Porting the Variant Calling Pipeline for NGS data in cloud-HPC environment},
author = {Alberto Mulone and Sherine Awad and Davide Chiarugi and Marco Aldinucci},
editor = {Hossain Shahriar and Yuuichi Teranishi and Alfredo Cuzzocrea and Moushumi Sharmin and Dave Towey and A. K. M. Jahangir Alam Majumder and Hiroki Kashiwazaki and Ji-Jiang Yang and Michiharu Takemoto and Nazmus Sakib and Ryohei Banno and Sheikh Iqbal Ahamed},
url = {https://iris.unito.it/bitstream/2318/1919364/1/paper.pdf},
doi = {10.1109/COMPSAC57700.2023.00288},
year = {2023},
date = {2023-01-01},
booktitle = {47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023},
pages = {1858–1863},
publisher = {IEEE},
address = {Torino, Italy},
abstract = {In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.},
howpublished = {47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023},
keywords = {across, icsc, streamflow},
pubstate = {published},
tppubtype = {inproceedings}
}
In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.
Iacopo Colonnelli, Bruno Casella, Gianluca Mittone, Yasir Arfat, Barbara Cantalupo, Roberto Esposito, Alberto Riccardo Martinelli, Doriana Medić, Marco Aldinucci
Federated Learning meets HPC and cloud Proceedings Article
In: Bufano, Filomena, Riggi, Simone, Sciacca, Eva, Schillirò, Francesco (Ed.): Astrophysics and Space Science Proceedings, pp. 193–199, Springer, Catania, Italy, 2023, ISBN: 978-3-031-34167-0, (Keynote talk).
Abstract | Links | BibTeX | Tags: across, eupilot, streamflow
@inproceedings{22:ml4astro,
title = {Federated Learning meets HPC and cloud},
author = {Iacopo Colonnelli and Bruno Casella and Gianluca Mittone and Yasir Arfat and Barbara Cantalupo and Roberto Esposito and Alberto Riccardo Martinelli and Doriana Medić and Marco Aldinucci},
editor = {Filomena Bufano and Simone Riggi and Eva Sciacca and Francesco Schillirò},
url = {https://iris.unito.it/retrieve/3ac66baa-9d9a-4e9f-94a5-13700694d8aa/ML4Astro.pdf},
doi = {10.1007/978-3-031-34167-0_39},
isbn = {978-3-031-34167-0},
year = {2023},
date = {2023-01-01},
booktitle = {Astrophysics and Space Science Proceedings},
volume = {60},
pages = {193–199},
publisher = {Springer},
address = {Catania, Italy},
abstract = {HPC and AI are fated to meet for several reasons. This article will discuss some of them and argue why this will happen through the set of methods and technologies that underpin cloud computing. As a paradigmatic example, we present a new federated learning system that collaboratively trains a deep learning model in different supercomputing centers. The system is based on the StreamFlow workflow manager designed for hybrid cloud-HPC infrastructures.},
howpublished = {Machine Learning for Astrophysics (ML4ASTRO)},
note = {Keynote talk},
keywords = {across, eupilot, streamflow},
pubstate = {published},
tppubtype = {inproceedings}
}
HPC and AI are fated to meet for several reasons. This article will discuss some of them and argue why this will happen through the set of methods and technologies that underpin cloud computing. As a paradigmatic example, we present a new federated learning system that collaboratively trains a deep learning model in different supercomputing centers. The system is based on the StreamFlow workflow manager designed for hybrid cloud-HPC infrastructures.
Sandro Gepiro Contaldo, Luca Alessandri, Iacopo Colonnelli, Marco Beccuti, Marco Aldinucci
Bringing Cell Subpopulation Discovery on a Cloud-HPC Using rCASC and StreamFlow Book Chapter
In: Calogero, Raffaele Adolfo, Benes, Vladimir (Ed.): Single Cell Transcriptomics: Methods and Protocols, pp. 337–345, Springer US, New York, NY, 2023, ISBN: 978-1-0716-2756-3.
Abstract | Links | BibTeX | Tags: streamflow
@inbook{Contaldo2023,
title = {Bringing Cell Subpopulation Discovery on a Cloud-HPC Using rCASC and StreamFlow},
author = {Sandro Gepiro Contaldo and Luca Alessandri and Iacopo Colonnelli and Marco Beccuti and Marco Aldinucci},
editor = {Raffaele Adolfo Calogero and Vladimir Benes},
url = {https://datacloud.di.unito.it/index.php/s/KMfKo4m7GTGdZmF},
doi = {10.1007/978-1-0716-2756-3_17},
isbn = {978-1-0716-2756-3},
year = {2023},
date = {2023-01-01},
booktitle = {Single Cell Transcriptomics: Methods and Protocols},
pages = {337–345},
publisher = {Springer US},
address = {New York, NY},
abstract = {The idea behind novel single-cell RNA sequencing (scRNA-seq) pipelines is to isolate single cells through microfluidic approaches and generate sequencing libraries in which the transcripts are tagged to track their cell of origin. Modern scRNA-seq platforms are capable of analyzing up to many thousands of cells in each run. Then, combined with massive high-throughput sequencing producing billions of reads, scRNA-seq allows the assessment of fundamental biological properties of cell populations and biological systems at unprecedented resolution.},
keywords = {streamflow},
pubstate = {published},
tppubtype = {inbook}
}
The idea behind novel single-cell RNA sequencing (scRNA-seq) pipelines is to isolate single cells through microfluidic approaches and generate sequencing libraries in which the transcripts are tagged to track their cell of origin. Modern scRNA-seq platforms are capable of analyzing up to many thousands of cells in each run. Then, combined with massive high-throughput sequencing producing billions of reads, scRNA-seq allows the assessment of fundamental biological properties of cell populations and biological systems at unprecedented resolution.
2022
Marco Aldinucci, David Atienza, Federico Bolelli, Mónica Caballero, Iacopo Colonnelli, José Flich, Jon Ander Gómez, David González, Costantino Grana, Marco Grangetto, Simone Leo, Pedro López, Dana Oniga, Roberto Paredes, Luca Pireddu, Eduardo Quiñones, Tatiana Silva, Enzo Tartaglione, Marina Zapater
In: Curry, Edward, Auer, Sören, Berre, Arne J., Metzger, Andreas, Perez, Maria S., Zillner, Sonja (Ed.): Technologies and Applications for Big Data Value, pp. 183–202, Springer International Publishing, Cham, 2022, ISBN: 978-3-030-78307-5.
Abstract | Links | BibTeX | Tags: deephealth, streamflow
@incollection{22:TABDV,
title = {The DeepHealth Toolkit: A Key European Free and Open-Source Software for Deep Learning and Computer Vision Ready to Exploit Heterogeneous HPC and Cloud Architectures},
author = {Marco Aldinucci and David Atienza and Federico Bolelli and Mónica Caballero and Iacopo Colonnelli and José Flich and Jon Ander Gómez and David González and Costantino Grana and Marco Grangetto and Simone Leo and Pedro López and Dana Oniga and Roberto Paredes and Luca Pireddu and Eduardo Quiñones and Tatiana Silva and Enzo Tartaglione and Marina Zapater},
editor = {Edward Curry and Sören Auer and Arne J. Berre and Andreas Metzger and Maria S. Perez and Sonja Zillner},
url = {https://link.springer.com/content/pdf/10.1007/978-3-030-78307-5_9.pdf},
doi = {10.1007/978-3-030-78307-5_9},
isbn = {978-3-030-78307-5},
year = {2022},
date = {2022-01-01},
booktitle = {Technologies and Applications for Big Data Value},
pages = {183–202},
publisher = {Springer International Publishing},
address = {Cham},
chapter = {9},
abstract = {At the present time, we are immersed in the convergence between Big Data, High-Performance Computing and Artificial Intelligence. Technological progress in these three areas has accelerated in recent years, forcing different players like software companies and stakeholders to move quickly. The European Union is dedicating a lot of resources to maintain its relevant position in this scenario, funding projects to implement large-scale pilot testbeds that combine the latest advances in Artificial Intelligence, High-Performance Computing, Cloud and Big Data technologies. The DeepHealth project is an example focused on the health sector whose main outcome is the DeepHealth toolkit, a European unified framework that offers deep learning and computer vision capabilities, completely adapted to exploit underlying heterogeneous High-Performance Computing, Big Data and cloud architectures, and ready to be integrated into any software platform to facilitate the development and deployment of new applications for specific problems in any sector. This toolkit is intended to be one of the European contributions to the field of AI. This chapter introduces the toolkit with its main components and complementary tools, providing a clear view to facilitate and encourage its adoption and wide use by the European community of developers of AI-based solutions and data scientists working in the healthcare sector and others.},
keywords = {deephealth, streamflow},
pubstate = {published},
tppubtype = {incollection}
}
At the present time, we are immersed in the convergence between Big Data, High-Performance Computing and Artificial Intelligence. Technological progress in these three areas has accelerated in recent years, forcing different players like software companies and stakeholders to move quickly. The European Union is dedicating a lot of resources to maintain its relevant position in this scenario, funding projects to implement large-scale pilot testbeds that combine the latest advances in Artificial Intelligence, High-Performance Computing, Cloud and Big Data technologies. The DeepHealth project is an example focused on the health sector whose main outcome is the DeepHealth toolkit, a European unified framework that offers deep learning and computer vision capabilities, completely adapted to exploit underlying heterogeneous High-Performance Computing, Big Data and cloud architectures, and ready to be integrated into any software platform to facilitate the development and deployment of new applications for specific problems in any sector. This toolkit is intended to be one of the European contributions to the field of AI. This chapter introduces the toolkit with its main components and complementary tools, providing a clear view to facilitate and encourage its adoption and wide use by the European community of developers of AI-based solutions and data scientists working in the healthcare sector and others.
Eduardo Quiñones, Jesus Perales, Jorge Ejarque, Asaf Badouh, Santiago Marco, Fabrice Auzanneau, François Galea, David González, José Ramón Hervás, Tatiana Silva, Iacopo Colonnelli, Barbara Cantalupo, Marco Aldinucci, Enzo Tartaglione, Rafael Tornero, José Flich, Jose Maria Martinez, David Rodriguez, Izan Catalán, Jorge Garcia, Carles Hernández
In: Terzo, Olivier, Martinovič, Jan (Ed.): HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision, pp. 191–216, CRC Press, Boca Raton, Florida, 2022, ISBN: 978-1-0320-0984-1.
Abstract | Links | BibTeX | Tags: deephealth, streamflow
@incollection{22:deephealth:HPCbook,
title = {The DeepHealth HPC Infrastructure: Leveraging Heterogenous HPC and Cloud Computing Infrastructures for IA-based Medical Solutions},
author = {Eduardo Quiñones and Jesus Perales and Jorge Ejarque and Asaf Badouh and Santiago Marco and Fabrice Auzanneau and François Galea and David González and José Ramón Hervás and Tatiana Silva and Iacopo Colonnelli and Barbara Cantalupo and Marco Aldinucci and Enzo Tartaglione and Rafael Tornero and José Flich and Jose Maria Martinez and David Rodriguez and Izan Catalán and Jorge Garcia and Carles Hernández},
editor = {Olivier Terzo and Jan Martinovič},
url = {https://iris.unito.it/retrieve/handle/2318/1832050/912413/Preprint.pdf},
doi = {10.1201/9781003176664},
isbn = {978-1-0320-0984-1},
year = {2022},
date = {2022-01-01},
booktitle = {HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision},
pages = {191–216},
publisher = {CRC Press},
address = {Boca Raton, Florida},
chapter = {10},
abstract = {This chapter presents the DeepHealth HPC toolkit for an efficient execution of deep learning (DL) medical application into HPC and cloud-computing infrastructures, featuring many-core, GPU, and FPGA acceleration devices. The toolkit offers to the European Computer Vision Library and the European Distributed Deep Learning Library (EDDL), developed in the DeepHealth project as well, the mechanisms to distribute and parallelize DL operations on HPC and cloud infrastructures in a fully transparent way. The toolkit implements workflow managers used to orchestrate HPC workloads for an efficient parallelization of EDDL training operations on HPC and cloud infrastructures, and includes the parallel programming models for an efficient execution EDDL inference and training operations on many-core, GPUs and FPGAs acceleration devices.},
keywords = {deephealth, streamflow},
pubstate = {published},
tppubtype = {incollection}
}
This chapter presents the DeepHealth HPC toolkit for an efficient execution of deep learning (DL) medical application into HPC and cloud-computing infrastructures, featuring many-core, GPU, and FPGA acceleration devices. The toolkit offers to the European Computer Vision Library and the European Distributed Deep Learning Library (EDDL), developed in the DeepHealth project as well, the mechanisms to distribute and parallelize DL operations on HPC and cloud infrastructures in a fully transparent way. The toolkit implements workflow managers used to orchestrate HPC workloads for an efficient parallelization of EDDL training operations on HPC and cloud infrastructures, and includes the parallel programming models for an efficient execution EDDL inference and training operations on many-core, GPUs and FPGAs acceleration devices.
Martin Golasowski, Jan Martinovič, Marc Levrier, Stephan Hachinger, Sophia Karagiorgou, Aikaterini Papapostolou, Spiros Mouzakitis, Ioannis Tsapelas, Monica Caballero, Marco Aldinucci, Jon Ander Gómez, Antony Chazapis, Jean-Thomas Acquaviva
Toward the Convergence of High-Performance Computing, Cloud, and Big Data Domains Book Section
In: Terzo, Olivier, Martinovič, Jan (Ed.): HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision, pp. 1–16, CRC Press, Boca Raton, Florida, 2022, ISBN: 978-1-0320-0984-1.
Abstract | Links | BibTeX | Tags: deephealth, streamflow
@incollection{22:intro:HPCbook,
title = {Toward the Convergence of High-Performance Computing, Cloud, and Big Data Domains},
author = {Martin Golasowski and Jan Martinovič and Marc Levrier and Stephan Hachinger and Sophia Karagiorgou and Aikaterini Papapostolou and Spiros Mouzakitis and Ioannis Tsapelas and Monica Caballero and Marco Aldinucci and Jon Ander Gómez and Antony Chazapis and Jean-Thomas Acquaviva},
editor = {Olivier Terzo and Jan Martinovič},
doi = {10.1201/9781003176664},
isbn = {978-1-0320-0984-1},
year = {2022},
date = {2022-01-01},
booktitle = {HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision},
pages = {1–16},
publisher = {CRC Press},
address = {Boca Raton, Florida},
chapter = {1},
abstract = {Convergence between big data, high-performance computing, and the cloud is the key driving factor for sustainable economic growth in the future. Technological advances in many fields are determined by competence to gain precise information from the large amounts of data collected, which in turn requires powerful computing resources. This chapter provides an overview on the evolution of the three fields and four different points of view on their convergence provided by the CYBELE, DeepHealth, Evolve, and LEXIS projects funded by the European Union under the Horizon 2020 Programme.},
keywords = {deephealth, streamflow},
pubstate = {published},
tppubtype = {incollection}
}
Convergence between big data, high-performance computing, and the cloud is the key driving factor for sustainable economic growth in the future. Technological advances in many fields are determined by competence to gain precise information from the large amounts of data collected, which in turn requires powerful computing resources. This chapter provides an overview on the evolution of the three fields and four different points of view on their convergence provided by the CYBELE, DeepHealth, Evolve, and LEXIS projects funded by the European Union under the Horizon 2020 Programme.
Dana Oniga, Barbara Cantalupo, Enzo Tartaglione, Daniele Perlo, Marco Grangetto, Marco Aldinucci, Federico Bolelli, Federico Pollastri, Michele Cancilla, Laura Canalini, Costantino Grana, Cristina Muñoz Alcalde, Franco Alberto Cardillo, Monica Florea
Applications of AI and HPC in the Health Domain Book Section
In: Terzo, Olivier, Martinovič, Jan (Ed.): HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision, pp. 217–239, CRC Press, Boca Raton, Florida, 2022, ISBN: 978-1-0320-0984-1.
Abstract | Links | BibTeX | Tags: deephealth, streamflow
@incollection{22:applications:HPCbook,
title = {Applications of AI and HPC in the Health Domain},
author = {Dana Oniga and Barbara Cantalupo and Enzo Tartaglione and Daniele Perlo and Marco Grangetto and Marco Aldinucci and Federico Bolelli and Federico Pollastri and Michele Cancilla and Laura Canalini and Costantino Grana and Cristina Muñoz Alcalde and Franco Alberto Cardillo and Monica Florea},
editor = {Olivier Terzo and Jan Martinovič},
doi = {10.1201/9781003176664},
isbn = {978-1-0320-0984-1},
year = {2022},
date = {2022-01-01},
booktitle = {HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision},
pages = {217–239},
publisher = {CRC Press},
address = {Boca Raton, Florida},
chapter = {11},
abstract = {This chapter presents the applications of artificial intelligence (AI) and high-computing performance (HPC) in the health domain, illustrated by the description of five of the use cases that are developed in the DeepHealth project. In the context of the European Commission supporting the use of AI and HPC in the health sector, DeepHealth Project is helping health experts process large quantities of images, putting at their disposal DeepLearning and computer vision techniques, combined in the DeepHealth toolkit and HPC infrastructures. The DeepHealth toolkit is tested and validated through 15 use cases, each of them representing a biomedical application. The most promising use cases are described in the chapter, which concludes with the value proposition and the benefits that DeepHealth toolkit offers to future end users.},
keywords = {deephealth, streamflow},
pubstate = {published},
tppubtype = {incollection}
}
This chapter presents the applications of artificial intelligence (AI) and high-computing performance (HPC) in the health domain, illustrated by the description of five of the use cases that are developed in the DeepHealth project. In the context of the European Commission supporting the use of AI and HPC in the health sector, DeepHealth Project is helping health experts process large quantities of images, putting at their disposal DeepLearning and computer vision techniques, combined in the DeepHealth toolkit and HPC infrastructures. The DeepHealth toolkit is tested and validated through 15 use cases, each of them representing a biomedical application. The most promising use cases are described in the chapter, which concludes with the value proposition and the benefits that DeepHealth toolkit offers to future end users.
Iacopo Colonnelli, Marco Aldinucci, Barbara Cantalupo, Luca Padovani, Sergio Rabellino, Concetto Spampinato, Roberto Morelli, Rosario Di Carlo, Nicolò Magini, Carlo Cavazzoni
Distributed workflows with Jupyter Journal Article
In: Future Generation Computer Systems, vol. 128, pp. 282–298, 2022, ISSN: 0167-739X.
Abstract | Links | BibTeX | Tags: across, deephealth, jupyter-workflow, streamflow
@article{21:FGCS:jupyflow,
title = {Distributed workflows with Jupyter},
author = {Iacopo Colonnelli and Marco Aldinucci and Barbara Cantalupo and Luca Padovani and Sergio Rabellino and Concetto Spampinato and Roberto Morelli and Rosario Di Carlo and Nicolò Magini and Carlo Cavazzoni},
url = {https://www.sciencedirect.com/science/article/pii/S0167739X21003976},
doi = {10.1016/j.future.2021.10.007},
issn = {0167-739X},
year = {2022},
date = {2022-01-01},
journal = {Future Generation Computer Systems},
volume = {128},
pages = {282–298},
abstract = {The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code's business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks' potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.},
keywords = {across, deephealth, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {article}
}
The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code's business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks' potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.
2021
Giovanni Agosta, William Fornaciari, Andrea Galimberti, Giuseppe Massari, Federico Reghenzani, Federico Terraneo, Davide Zoni, Carlo Brandolese, Massimo Celino, Francesco Iannone, Paolo Palazzari, Giuseppe Zummo, Massimo Bernaschi, Pasqua D'Ambra, Sergio Saponara, Marco Danelutto, Massimo Torquati, Marco Aldinucci, Yasir Arfat, Barbara Cantalupo, Iacopo Colonnelli, Roberto Esposito, Alberto Riccardo Martinelli, Gianluca Mittone, Olivier Beaumont, Berenger Bramas, Lionel Eyraud-Dubois, Brice Goglin, Abdou Guermouche, Raymond Namyst, Samuel Thibault, Antonio Filgueras, Miquel Vidal, Carlos Alvarez, Xavier Martorell, Ariel Oleksiak, Michal Kulczewski, Alessandro Lonardo, Piero Vicini, Francesco Lo Cicero, Francesco Simula, Andrea Biagioni, Paolo Cretaro, Ottorino Frezza, Pier Stanislao Paolucci, Matteo Turisini, Francesco Giacomini, Tommaso Boccali, Simone Montangero, Roberto Ammendola
TEXTAROSSA: Towards EXtreme scale Technologies and Accelerators for euROhpc hw/Sw Supercomputing Applications for exascale Proceedings Article
In: Proc. of the 24th Euromicro Conference on Digital System Design (DSD), IEEE, Palermo, Italy, 2021.
Abstract | Links | BibTeX | Tags: streamflow, textarossa
@inproceedings{21:DSD:textarossa,
title = {TEXTAROSSA: Towards EXtreme scale Technologies and Accelerators for euROhpc hw/Sw Supercomputing Applications for exascale},
author = {Giovanni Agosta and William Fornaciari and Andrea Galimberti and Giuseppe Massari and Federico Reghenzani and Federico Terraneo and Davide Zoni and Carlo Brandolese and Massimo Celino and Francesco Iannone and Paolo Palazzari and Giuseppe Zummo and Massimo Bernaschi and Pasqua D'Ambra and Sergio Saponara and Marco Danelutto and Massimo Torquati and Marco Aldinucci and Yasir Arfat and Barbara Cantalupo and Iacopo Colonnelli and Roberto Esposito and Alberto Riccardo Martinelli and Gianluca Mittone and Olivier Beaumont and Berenger Bramas and Lionel Eyraud-Dubois and Brice Goglin and Abdou Guermouche and Raymond Namyst and Samuel Thibault and Antonio Filgueras and Miquel Vidal and Carlos Alvarez and Xavier Martorell and Ariel Oleksiak and Michal Kulczewski and Alessandro Lonardo and Piero Vicini and Francesco Lo Cicero and Francesco Simula and Andrea Biagioni and Paolo Cretaro and Ottorino Frezza and Pier Stanislao Paolucci and Matteo Turisini and Francesco Giacomini and Tommaso Boccali and Simone Montangero and Roberto Ammendola},
doi = {10.1109/DSD53832.2021.00051},
year = {2021},
date = {2021-08-01},
booktitle = {Proc. of the 24th Euromicro Conference on Digital System Design (DSD)},
publisher = {IEEE},
address = {Palermo, Italy},
abstract = {To achieve high performance and high energy effi- ciency on near-future exascale computing systems, three key technology gaps needs to be bridged. These gaps include: en- ergy efficiency and thermal control; extreme computation effi- ciency via HW acceleration and new arithmetics; methods and tools for seamless integration of reconfigurable accelerators in heterogeneous HPC multi-node platforms. TEXTAROSSA aims at tackling this gap through a co-design approach to heterogeneous HPC solutions, supported by the integration and extension of HW and SW IPs, programming models and tools derived from European research.},
keywords = {streamflow, textarossa},
pubstate = {published},
tppubtype = {inproceedings}
}
To achieve high performance and high energy effi- ciency on near-future exascale computing systems, three key technology gaps needs to be bridged. These gaps include: en- ergy efficiency and thermal control; extreme computation effi- ciency via HW acceleration and new arithmetics; methods and tools for seamless integration of reconfigurable accelerators in heterogeneous HPC multi-node platforms. TEXTAROSSA aims at tackling this gap through a co-design approach to heterogeneous HPC solutions, supported by the integration and extension of HW and SW IPs, programming models and tools derived from European research.
Iacopo Colonnelli, Barbara Cantalupo, Concetto Spampinato, Matteo Pennisi, Marco Aldinucci
Bringing AI pipelines onto cloud-HPC: setting a baseline for accuracy of COVID-19 diagnosis Proceedings Article
In: Iannone, Francesco (Ed.): ENEA CRESCO in the fight against COVID-19, ENEA, 2021.
Abstract | Links | BibTeX | Tags: streamflow
@inproceedings{21:covi:enea,
title = {Bringing AI pipelines onto cloud-HPC: setting a baseline for accuracy of COVID-19 diagnosis},
author = {Iacopo Colonnelli and Barbara Cantalupo and Concetto Spampinato and Matteo Pennisi and Marco Aldinucci},
editor = {Francesco Iannone},
url = {https://iris.unito.it/retrieve/handle/2318/1796029/779853/21_AI-pipelines_ENEA-COVID19.pdf},
doi = {10.5281/zenodo.5151511},
year = {2021},
date = {2021-01-01},
booktitle = {ENEA CRESCO in the fight against COVID-19},
publisher = {ENEA},
abstract = {HPC is an enabling platform for AI. The introduction of AI workloads in the HPC applications basket has non-trivial consequences both on the way of designing AI applications and on the way of providing HPC computing. This is the leitmotif of the convergence between HPC and AI. The formalized definition of AI pipelines is one of the milestones of HPC-AI convergence. If well conducted, it allows, on the one hand, to obtain portable and scalable applications. On the other hand, it is crucial for the reproducibility of scientific pipelines. In this work, we advocate the StreamFlow Workflow Management System as a crucial ingredient to define a parametric pipeline, called ``CLAIRE COVID-19 Universal Pipeline'', which is able to explore the optimization space of methods to classify COVID-19 lung lesions from CT scans, compare them for accuracy, and therefore set a performance baseline. The universal pipeline automatizes the training of many different Deep Neural Networks (DNNs) and many different hyperparameters. It, therefore, requires a massive computing power, which is found in traditional HPC infrastructure thanks to the portability-by-design of pipelines designed with StreamFlow. Using the universal pipeline, we identified a DNN reaching over 90% accuracy in detecting COVID-19 lesions in CT scans.},
keywords = {streamflow},
pubstate = {published},
tppubtype = {inproceedings}
}
HPC is an enabling platform for AI. The introduction of AI workloads in the HPC applications basket has non-trivial consequences both on the way of designing AI applications and on the way of providing HPC computing. This is the leitmotif of the convergence between HPC and AI. The formalized definition of AI pipelines is one of the milestones of HPC-AI convergence. If well conducted, it allows, on the one hand, to obtain portable and scalable applications. On the other hand, it is crucial for the reproducibility of scientific pipelines. In this work, we advocate the StreamFlow Workflow Management System as a crucial ingredient to define a parametric pipeline, called ``CLAIRE COVID-19 Universal Pipeline'', which is able to explore the optimization space of methods to classify COVID-19 lung lesions from CT scans, compare them for accuracy, and therefore set a performance baseline. The universal pipeline automatizes the training of many different Deep Neural Networks (DNNs) and many different hyperparameters. It, therefore, requires a massive computing power, which is found in traditional HPC infrastructure thanks to the portability-by-design of pipelines designed with StreamFlow. Using the universal pipeline, we identified a DNN reaching over 90% accuracy in detecting COVID-19 lesions in CT scans.
Iacopo Colonnelli, Barbara Cantalupo, Roberto Esposito, Matteo Pennisi, Concetto Spampinato, Marco Aldinucci
HPC Application Cloudification: The StreamFlow Toolkit Proceedings Article
In: Bispo, João, Cherubin, Stefano, Flich, José (Ed.): 12th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures and 10th Workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms (PARMA-DITAM 2021), pp. 5:1–5:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 2021, ISSN: 2190-6807.
Abstract | Links | BibTeX | Tags: deephealth, hpc4ai, streamflow
@inproceedings{colonnelli_et_al:OASIcs.PARMA-DITAM.2021.5,
title = {HPC Application Cloudification: The StreamFlow Toolkit},
author = {Iacopo Colonnelli and Barbara Cantalupo and Roberto Esposito and Matteo Pennisi and Concetto Spampinato and Marco Aldinucci},
editor = {João Bispo and Stefano Cherubin and José Flich},
url = {https://drops.dagstuhl.de/opus/volltexte/2021/13641/pdf/OASIcs-PARMA-DITAM-2021-5.pdf},
doi = {10.4230/OASIcs.PARMA-DITAM.2021.5},
issn = {2190-6807},
year = {2021},
date = {2021-01-01},
booktitle = {12th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures and 10th Workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms (PARMA-DITAM 2021)},
volume = {88},
pages = {5:1–5:13},
publisher = {Schloss Dagstuhl – Leibniz-Zentrum für Informatik},
address = {Dagstuhl, Germany},
series = {Open Access Series in Informatics (OASIcs)},
abstract = {Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud infrastructure can be offloaded to HPC environments to speed them up. We introduce StreamFlow, a novel Workflow Management System that supports such a design pattern and makes it possible to run the steps of a standard workflow model on independent processing elements with no shared storage. We validated the proposed approach's effectiveness on the CLAIRE COVID-19 universal pipeline, i.e. a reproducible workflow capable of automating the comparison of (possibly all) state-of-the-art pipelines for the diagnosis of COVID-19 interstitial pneumonia from CT scans images based on Deep Neural Networks (DNNs).},
keywords = {deephealth, hpc4ai, streamflow},
pubstate = {published},
tppubtype = {inproceedings}
}
Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud Finding an effective way to improve accessibility to High-Performance Computing facilities, still anchored to SSH-based remote shells and queue-based job submission mechanisms, is an open problem in computer science. This work advocates a cloudification of HPC applications through a cluster-as-accelerator pattern, where computationally demanding portions of the main execution flow hosted on a Cloud infrastructure can be offloaded to HPC environments to speed them up. We introduce StreamFlow, a novel Workflow Management System that supports such a design pattern and makes it possible to run the steps of a standard workflow model on independent processing elements with no shared storage. We validated the proposed approach's effectiveness on the CLAIRE COVID-19 universal pipeline, i.e. a reproducible workflow capable of automating the comparison of (possibly all) state-of-the-art pipelines for the diagnosis of COVID-19 interstitial pneumonia from CT scans images based on Deep Neural Networks (DNNs).
Iacopo Colonnelli, Barbara Cantalupo, Ivan Merelli, Marco Aldinucci
StreamFlow: cross-breeding cloud with HPC Journal Article
In: IEEE Transactions on Emerging Topics in Computing, vol. 9, no. 4, pp. 1723–1737, 2021.
Abstract | Links | BibTeX | Tags: deephealth, hpc4ai, streamflow
@article{20Lstreamflow:tetc,
title = {StreamFlow: cross-breeding cloud with HPC},
author = {Iacopo Colonnelli and Barbara Cantalupo and Ivan Merelli and Marco Aldinucci},
url = {https://arxiv.org/pdf/2002.01558},
doi = {10.1109/TETC.2020.3019202},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Emerging Topics in Computing},
volume = {9},
number = {4},
pages = {1723–1737},
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single cell transcriptomic data analysis workflow.},
keywords = {deephealth, hpc4ai, streamflow},
pubstate = {published},
tppubtype = {article}
}
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single cell transcriptomic data analysis workflow.
Talks
2024
Alberto Mulone, Doriana Medić, Marco Aldinucci
A Fault Tolerance mechanism for Hybrid Scientific Workflows Miscellaneous
1st workshop about High-Performance e-Science (HiPES), 2024.
Abstract | Links | BibTeX | Tags: eupex, icsc, streamflow
@misc{24:madrid:hipes:talk,
title = {A Fault Tolerance mechanism for Hybrid Scientific Workflows},
author = {Alberto Mulone and Doriana Medić and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/9Ddj6fGgmDbLDXj},
year = {2024},
date = {2024-08-01},
address = {Madrid, Spain},
abstract = {In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.},
howpublished = {1st workshop about High-Performance e-Science (HiPES)},
keywords = {eupex, icsc, streamflow},
pubstate = {published},
tppubtype = {misc}
}
In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.
Iacopo Colonnelli
Dynamic hybrid workflows for Deep Learning on HPC infrastructure Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: icsc, jupyter-workflow, streamflow
@misc{24:icolonne:ictp,
title = {Dynamic hybrid workflows for Deep Learning on HPC infrastructure},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/EaFHJEKNbW5oXeq},
year = {2024},
date = {2024-05-01},
address = {Trieste, Italy},
abstract = {Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher’s laptop to an entire HPC facility.},
keywords = {icsc, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Hybrid workflow abstractions allow users to quickly design and orchestrate cross-facility workloads, decoupling tasks from environment-specific technical details to reduce complexity and increase reusability. Plus, workflow descriptions help ensure the reproducibility of scientific experiments through prospective and retrospective provenance collection. This module has been designed to provide a hands-on exploration of scientific workflows from various angles, from the initial design phase to their orchestration at extreme scales. We will use the practical example of the CommonWorkflow Language (CWL) open standard to demonstrate how workflows can be written, and the StreamFlow workflow system to execute them seamlessly on the CINECA HPC facility. We will also delve into the integration between scientific workflows and Jupyter Notebooks, which aims to give data scientists a familiar interface to scientific workflows. In this module, students will gain a comprehensive understanding of scientific workflows. They will learn how to use these workflows to model and orchestrate Machine Learning and Deep Learning pipelines. Additionally, they will explore how modern workflow management systems can efficiently scale data-oriented workloads from a researcher’s laptop to an entire HPC facility.
Iacopo Colonnelli
CWL in the HPC Ecosystem Miscellaneous
Workshop on workflow languages for HEP analysis, 2024.
Links | BibTeX | Tags: across, eupex, icsc, space, streamflow
@misc{24:icolonne:cwl4hpccern,
title = {CWL in the HPC Ecosystem},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/PRmqdwWHt6P2PH7},
year = {2024},
date = {2024-04-01},
address = {CERN, Meyrin, Switzerland},
howpublished = {Workshop on workflow languages for HEP analysis},
keywords = {across, eupex, icsc, space, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Alberto Mulone
Cross-Platform Full Waveform Inversion Miscellaneous
ICSC - Spoke 1 meeting, 2024.
Abstract | Links | BibTeX | Tags: icsc, streamflow
@misc{24:icsc:spoke1:eni,
title = {Cross-Platform Full Waveform Inversion},
author = {Alberto Mulone},
url = {https://datacloud.di.unito.it/index.php/s/M3HkxA5wsBPS5ro},
year = {2024},
date = {2024-02-01},
address = {Torino, Italy},
abstract = {Presentation about the Innovation Grant in collaboration with ENI},
howpublished = {ICSC - Spoke 1 meeting},
keywords = {icsc, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Presentation about the Innovation Grant in collaboration with ENI
2023
Alberto Scionti, Iacopo Colonnelli
Orchestrating Multi-Domain Workflows: The ACROSS Approach Miscellaneous
Workflows Community: Modern Workflows for Continuum and Cross-Facility Computing, 2023.
Links | BibTeX | Tags: across, streamflow
@misc{23:sc:WCIBoF,
title = {Orchestrating Multi-Domain Workflows: The ACROSS Approach},
author = {Alberto Scionti and Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/rJXcDBK4mLmS8yz},
year = {2023},
date = {2023-11-01},
address = {Denver, CO, Usa},
howpublished = {Workflows Community: Modern Workflows for Continuum and Cross-Facility Computing},
keywords = {across, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
ACROSS: HPC Big Data Artificial Intelligence Cross Stack Platform Towards Exascale Miscellaneous
LN HPC-KTT Assemblea Nazionale 2023, 2023.
Links | BibTeX | Tags: across, streamflow
@misc{23:AssembleaHPC-KTT,
title = {ACROSS: HPC Big Data Artificial Intelligence Cross Stack Platform Towards Exascale},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/aK7es8BgFeWorjD},
year = {2023},
date = {2023-10-01},
address = {Pisa, Italy},
howpublished = {LN HPC-KTT Assemblea Nazionale 2023},
keywords = {across, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli, Doriana Medić, Barbara Cantalupo, Marco Aldinucci
Università degli Studi di Torino: Alpha parallel research group Miscellaneous
HaMMon Kick-Off meeting, 2023.
Links | BibTeX | Tags: icsc, streamflow
@misc{23:HaMMonProject,
title = {Università degli Studi di Torino: Alpha parallel research group},
author = {Iacopo Colonnelli and Doriana Medić and Barbara Cantalupo and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/cmgy9BZ3nwCR2QJ},
year = {2023},
date = {2023-10-01},
address = {Bologna, Italy},
howpublished = {HaMMon Kick-Off meeting},
keywords = {icsc, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
Workflow models for heterogeneous distributed systems Miscellaneous
2nd Italian Conference on Big Data and Data Science (ITADATA 2023), 2023, (Best PhD Thesis Award).
Links | BibTeX | Tags: jupyter-workflow, streamflow
@misc{23:ITADATABestPhDThesis,
title = {Workflow models for heterogeneous distributed systems},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/6RqcaJ4djqFNDC8},
year = {2023},
date = {2023-09-01},
address = {Napoli, Italy},
howpublished = {2nd Italian Conference on Big Data and Data Science (ITADATA 2023)},
note = {Best PhD Thesis Award},
keywords = {jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Alberto Mulone, Sherine Awad, Davide Chiarugi, Marco Aldinucci
Porting the Variant Calling Pipeline for NGS data in cloud-HPC environment Miscellaneous
47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023, 2023.
Abstract | Links | BibTeX | Tags: across, icsc, streamflow
@misc{23:mulone:wide:talk,
title = {Porting the Variant Calling Pipeline for NGS data in cloud-HPC environment},
author = {Alberto Mulone and Sherine Awad and Davide Chiarugi and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/zNLj3LCZNsNxHwy},
year = {2023},
date = {2023-06-01},
address = {Torino, Italy},
abstract = {In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.},
howpublished = {47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023},
keywords = {across, icsc, streamflow},
pubstate = {published},
tppubtype = {misc}
}
In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.
Iacopo Colonnelli
Workflows and the Common Workflow Language (CWL) Miscellaneous
OSA2Micro: An Open Science Approach to Microbiology data integration, 2023, (Invited talk).
Links | BibTeX | Tags: streamflow
@misc{23:SA2Micro,
title = {Workflows and the Common Workflow Language (CWL)},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/NHWWzMMaQgAsA52},
year = {2023},
date = {2023-05-01},
address = {Torino, Italy},
howpublished = {OSA2Micro: An Open Science Approach to Microbiology data integration},
note = {Invited talk},
keywords = {streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
UNITO tools presentation Miscellaneous
CN HPC Flagship 3 Working Day, 2023.
Links | BibTeX | Tags: jupyter-workflow, streamflow
@misc{23:FL3WorkingDay,
title = {UNITO tools presentation},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/fgHbnLDQSFtcwLd},
year = {2023},
date = {2023-05-01},
address = {Bologna, Italy},
howpublished = {CN HPC Flagship 3 Working Day},
keywords = {jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Sofia Karvounari, Eleni Mathioulaki, Michael R. Crusoe, Iacopo Colonnelli
Standardised Workflows at EBRAINS Miscellaneous
Human Brain Project Summit 2023, 2023, (Invited talk).
Abstract | Links | BibTeX | Tags: across, eupex, space, streamflow
@misc{23:HBPSummit,
title = {Standardised Workflows at EBRAINS},
author = {Sofia Karvounari and Eleni Mathioulaki and Michael R. Crusoe and Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/K5YQKTsX9N7NLT8},
year = {2023},
date = {2023-03-01},
address = {Marseille, France},
abstract = {A hands-on training offer for Standardised Workflows in EBRAINS. A short presentation will be used as an introduction, while the main hands-on session will provide information about Writing and Executing Standardised Workflows. TC will give some guidelines, so attendees can experiment with writing CWL tools and workflows and then they will be given access to VM to execute these workflows. The Workflows Dashboard will be also presented during the same session, offering to the attendees the opportunity to understand the different functionalities, use it with TC support and provide useful comments.},
howpublished = {Human Brain Project Summit 2023},
note = {Invited talk},
keywords = {across, eupex, space, streamflow},
pubstate = {published},
tppubtype = {misc}
}
A hands-on training offer for Standardised Workflows in EBRAINS. A short presentation will be used as an introduction, while the main hands-on session will provide information about Writing and Executing Standardised Workflows. TC will give some guidelines, so attendees can experiment with writing CWL tools and workflows and then they will be given access to VM to execute these workflows. The Workflows Dashboard will be also presented during the same session, offering to the attendees the opportunity to understand the different functionalities, use it with TC support and provide useful comments.
Iacopo Colonnelli
CWL for HPC: are we there yet? Miscellaneous
2023 CWL Conference, 2023, (Invited talk).
Abstract | Links | BibTeX | Tags: across, eupex, streamflow
@misc{23:CWLConference,
title = {CWL for HPC: are we there yet?},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/CMCd5LiZeXsxwEg},
year = {2023},
date = {2023-03-01},
address = {Heidelberg, Germany},
abstract = {Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWL’s readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.},
howpublished = {2023 CWL Conference},
note = {Invited talk},
keywords = {across, eupex, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWL’s readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.
2022
Iacopo Colonnelli
StreamFlow Miscellaneous
2nd HealthyCloud Workshop: Analysis of existing orchestration mechanisms for distributed computational analyses, 2022, (Invited talk).
Links | BibTeX | Tags: across, deephealth, eupex, streamflow, textarossa
@misc{22:healthycloud-workshop,
title = {StreamFlow},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/Taz8qtzmkmn9ffT},
year = {2022},
date = {2022-07-01},
address = {Virtual event},
howpublished = {2nd HealthyCloud Workshop: Analysis of existing orchestration mechanisms for distributed computational analyses},
note = {Invited talk},
keywords = {across, deephealth, eupex, streamflow, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
StreamFlow: a topology-aware WMS Miscellaneous
ELIXIR Cloud, Data & AAI Bi-weekly Technical Calls, 2022, (Invited talk).
Links | BibTeX | Tags: across, deephealth, eupex, streamflow, textarossa
@misc{22:elixir-streamflow,
title = {StreamFlow: a topology-aware WMS},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/Z9GsKnRCxmBdMd3},
year = {2022},
date = {2022-06-01},
address = {Virtual event},
howpublished = {ELIXIR Cloud, Data & AAI Bi-weekly Technical Calls},
note = {Invited talk},
keywords = {across, deephealth, eupex, streamflow, textarossa},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
StreamFlow: A framework for hybrid workflows Miscellaneous
EUPEX WP5 bi-weekly meeting, 2022.
Links | BibTeX | Tags: eupex, streamflow
@misc{22:eupex-streamflow,
title = {StreamFlow: A framework for hybrid workflows},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/NjKEySP7HfrCQHZ},
year = {2022},
date = {2022-04-01},
address = {Virtual event},
howpublished = {EUPEX WP5 bi-weekly meeting},
keywords = {eupex, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli, Dario Tranchitella
OpenDeepHealth: Crafting a Deep Learning Platform as a Service with Kubernetes Miscellaneous
J on The Beach 2022, 2022.
Links | BibTeX | Tags: across, deephealth, hpc4ai, jupyter-workflow, streamflow
@misc{22:jotb22,
title = {OpenDeepHealth: Crafting a Deep Learning Platform as a Service with Kubernetes},
author = {Iacopo Colonnelli and Dario Tranchitella},
url = {https://datacloud.di.unito.it/index.php/s/n6J7STNnwdyqtET},
year = {2022},
date = {2022-04-01},
address = {Malaga, Spain},
howpublished = {J on The Beach 2022},
keywords = {across, deephealth, hpc4ai, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
Distributed workflows with Jupyter Miscellaneous
J on The Beach 2022, 2022, (Workshop).
Links | BibTeX | Tags: across, deephealth, jupyter-workflow, streamflow
@misc{22:jotb22-workshop,
title = {Distributed workflows with Jupyter},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/om89q55S6ePf2Ji},
year = {2022},
date = {2022-04-01},
address = {Malaga, Spain},
howpublished = {J on The Beach 2022},
note = {Workshop},
keywords = {across, deephealth, jupyter-workflow, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Iacopo Colonnelli
StreamFlow: A framework for hybrid workflows Miscellaneous
ACROSS WP4 meeting, 2022.
Links | BibTeX | Tags: across, streamflow
@misc{22:across-streamflow,
title = {StreamFlow: A framework for hybrid workflows},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/FXFTKtQSRf6anMX},
year = {2022},
date = {2022-02-01},
address = {Virtual event},
howpublished = {ACROSS WP4 meeting},
keywords = {across, streamflow},
pubstate = {published},
tppubtype = {misc}
}
2021
Iacopo Colonnelli
StreamFlow: A framework for hybrid workflows Miscellaneous
ACROSS WP4 meeting, 2021.
Links | BibTeX | Tags: across, streamflow
@misc{21:across-streamflow,
title = {StreamFlow: A framework for hybrid workflows},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/yrGYJL6CyNywF8a},
year = {2021},
date = {2021-10-01},
address = {Virtual event},
howpublished = {ACROSS WP4 meeting},
keywords = {across, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Marco Aldinucci
The modernization of HPC applications for the cloud era Miscellaneous
Fifth EAGE Workshop on High Performance Computing for Upstream, 2021, (Keynote talk).
Abstract | BibTeX | Tags: across, admire, deephealth, streamflow
@misc{21:eni:streamflow,
title = {The modernization of HPC applications for the cloud era},
author = {Marco Aldinucci},
year = {2021},
date = {2021-09-01},
address = {Virtual event},
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g., clouds, supercomputers, and both of them. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments (such as Kubernetes and SLURM), making it possible to execute onto multiple sites not sharing a common data space. Streamflow clearly distinguishes it from many other workflow management systems because it decouples the data dependencies from the deployment of (containerized) workflow steps. Streamflow also leverages CAPIO (Cross-Application Programmable I/O) to move data from one step to another efficiently. CAPIO captures the POSIX file system and streams it in parallel and in-memory to the workflow's next step, possibly enabling in-transit data filtering.},
howpublished = {Fifth EAGE Workshop on High Performance Computing for Upstream},
note = {Keynote talk},
keywords = {across, admire, deephealth, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g., clouds, supercomputers, and both of them. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments (such as Kubernetes and SLURM), making it possible to execute onto multiple sites not sharing a common data space. Streamflow clearly distinguishes it from many other workflow management systems because it decouples the data dependencies from the deployment of (containerized) workflow steps. Streamflow also leverages CAPIO (Cross-Application Programmable I/O) to move data from one step to another efficiently. CAPIO captures the POSIX file system and streams it in parallel and in-memory to the workflow's next step, possibly enabling in-transit data filtering.
Marco Aldinucci
From skeletons to workflows in the cloud-edge era Miscellaneous
14th Intl. Symposium on High-Level Programming and Applications (HLPP), 2021, (Keynote talk).
Abstract | Links | BibTeX | Tags: across, admire, deephealth, streamflow
@misc{21:hlpp:streamflow,
title = {From skeletons to workflows in the cloud-edge era},
author = {Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/RyRPjNBse5PKnab},
year = {2021},
date = {2021-07-01},
address = {Virtual event},
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments and that makes it possible to execute multiple sites not sharing a common data space. StreamFlow supports both task and data parallelism and enables the reproducible and scalable execution of workflows, such as AI pipelines, in hybrid cloud-HPC environments. As a running example, we use the novel ``universal COVID-19 pipeline'' that explore the whole optimisation space of the training of different DNNs to classify COVID-19 lung lesions.},
howpublished = {14th Intl. Symposium on High-Level Programming and Applications (HLPP)},
note = {Keynote talk},
keywords = {across, admire, deephealth, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments and that makes it possible to execute multiple sites not sharing a common data space. StreamFlow supports both task and data parallelism and enables the reproducible and scalable execution of workflows, such as AI pipelines, in hybrid cloud-HPC environments. As a running example, we use the novel ``universal COVID-19 pipeline'' that explore the whole optimisation space of the training of different DNNs to classify COVID-19 lung lesions.
Marco Aldinucci, Iacopo Colonnelli
The Universal Cloud-HPC Pipeline for the AI-Assisted Explainable Diagnosis of COVID-19 Pneumonia Miscellaneous
NVidia GTC'21, 2021, (Invited talk).
Abstract | Links | BibTeX | Tags: deephealth, hpc4ai, streamflow
@misc{21:gtc:clairecovid,
title = {The Universal Cloud-HPC Pipeline for the AI-Assisted Explainable Diagnosis of COVID-19 Pneumonia},
author = {Marco Aldinucci and Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/AkQLbPpEEtDzbbm},
year = {2021},
date = {2021-04-01},
address = {Virtual event},
abstract = {We'll present a methodology to run DNN pipelines on hybrid cloud+HPC infrastructure. We'll also define a "universal pipeline" for medical images. The pipeline can reproduce all state-of-the-art DNNs to diagnose COVID-19 pneumonia, which appeared in the literature during the first Italian lockdown and following months. We can run all of them (across cloud+HPC platforms) and compare their performance in terms of sensitivity and specificity to set a baseline to evaluate future progress in the automated diagnosis of COVID-19. Also, the pipeline makes existing DNNs explainable by way of adversarial training. The pipeline is easily portable and can run across different infrastructures, adapting the performance-urgency trade-off. The methodology builds onto two novel software programs: the streamflow workflow system and the AI-sandbox concept (parallel container with user-space encrypted file system). We reach over 92% accuracy in diagnosing COVID pneumonia.},
howpublished = {NVidia GTC'21},
note = {Invited talk},
keywords = {deephealth, hpc4ai, streamflow},
pubstate = {published},
tppubtype = {misc}
}
We'll present a methodology to run DNN pipelines on hybrid cloud+HPC infrastructure. We'll also define a "universal pipeline" for medical images. The pipeline can reproduce all state-of-the-art DNNs to diagnose COVID-19 pneumonia, which appeared in the literature during the first Italian lockdown and following months. We can run all of them (across cloud+HPC platforms) and compare their performance in terms of sensitivity and specificity to set a baseline to evaluate future progress in the automated diagnosis of COVID-19. Also, the pipeline makes existing DNNs explainable by way of adversarial training. The pipeline is easily portable and can run across different infrastructures, adapting the performance-urgency trade-off. The methodology builds onto two novel software programs: the streamflow workflow system and the AI-sandbox concept (parallel container with user-space encrypted file system). We reach over 92% accuracy in diagnosing COVID pneumonia.
Iacopo Colonnelli
StreamFlow: cross breeding cloud with HPC Miscellaneous
2021 CWL Mini Conference, 2021, (Invited talk).
Abstract | Links | BibTeX | Tags: deephealth, streamflow
@misc{21:CWLMiniConference,
title = {StreamFlow: cross breeding cloud with HPC},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/Le9gg4PfjRxBwXD},
year = {2021},
date = {2021-02-01},
address = {Virtual event},
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space.},
howpublished = {2021 CWL Mini Conference},
note = {Invited talk},
keywords = {deephealth, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space.
2020
Iacopo Colonnelli
StreamFlow: cross breeding cloud with HPC Miscellaneous
HPC-Europa3 2nd Transnational Access Meeting (TAM), 2020, (Invited talk).
Abstract | Links | BibTeX | Tags: streamflow
@misc{20:HPCEuropa3TAM,
title = {StreamFlow: cross breeding cloud with HPC},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/qPHHrSNxk8QXJDw},
year = {2020},
date = {2020-10-01},
address = {Virtual event},
abstract = {Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.},
howpublished = {HPC-Europa3 2nd Transnational Access Meeting (TAM)},
note = {Invited talk},
keywords = {streamflow},
pubstate = {published},
tppubtype = {misc}
}
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.
2019
Iacopo Colonnelli
StreamFlow: un approccio dichiarativo a workflow e pipeline di micro-servizi Miscellaneous
Workshop GARR 2019, 2019.
Abstract | Links | BibTeX | Tags: streamflow
@misc{19:GarrWorkshop,
title = {StreamFlow: un approccio dichiarativo a workflow e pipeline di micro-servizi},
author = {Iacopo Colonnelli},
url = {https://datacloud.di.unito.it/index.php/s/kZqyiQnBEQNdXJe},
year = {2019},
date = {2019-10-01},
address = {Roma, Italy},
abstract = {Negli ultimi anni, gli approcci orientati ai container si sono dimostrati particolarmente efficaci nel garantire portabilità e riproducibilità dei workflow scientifici. Tuttavia, con il continuo aumento del volume di dati a disposizione e la crescente complessità delle procedure di analisi in ogni campo della ricerca, anche i requisiti di performance e riusabilità si fanno via via sempre più essenziali. L'obiettivo principale di StreamFlow è quello di fornire un nuovo paradigma, totalmente dichiarativo, per la descrizione e l'accelerazione di workflow scientifici in ambienti distribuiti. La peculiarità di StreamFlow risiede nel fatto che l'ambiente di esecuzione è interamente descritto in termini di servizi (container), connessioni tra essi e fattori di replica. Inoltre, ogni task del workflow è esplicitamente mappato sulla tipologia di servizio richiesta. Questo permette un maggior controllo sull'utilizzo delle risorse e politiche di scheduling più precise, a vantaggio delle performance. I principali vantaggi di un approccio dichiarativo sono invece la più facile comprensione ed estensione dei modelli esistenti, a vantaggio della riusabilitià.},
howpublished = {Workshop GARR 2019},
keywords = {streamflow},
pubstate = {published},
tppubtype = {misc}
}
Negli ultimi anni, gli approcci orientati ai container si sono dimostrati particolarmente efficaci nel garantire portabilità e riproducibilità dei workflow scientifici. Tuttavia, con il continuo aumento del volume di dati a disposizione e la crescente complessità delle procedure di analisi in ogni campo della ricerca, anche i requisiti di performance e riusabilità si fanno via via sempre più essenziali. L'obiettivo principale di StreamFlow è quello di fornire un nuovo paradigma, totalmente dichiarativo, per la descrizione e l'accelerazione di workflow scientifici in ambienti distribuiti. La peculiarità di StreamFlow risiede nel fatto che l'ambiente di esecuzione è interamente descritto in termini di servizi (container), connessioni tra essi e fattori di replica. Inoltre, ogni task del workflow è esplicitamente mappato sulla tipologia di servizio richiesta. Questo permette un maggior controllo sull'utilizzo delle risorse e politiche di scheduling più precise, a vantaggio delle performance. I principali vantaggi di un approccio dichiarativo sono invece la più facile comprensione ed estensione dei modelli esistenti, a vantaggio della riusabilitià.
CAPIO
A joint project with CS Dept. – University of Pisa
CAPIO (Cross-Application Programmable I/O) is a middleware capable of transparently injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without modifying the business code. The contribution is twofold: at design time, a new I/O coordination language allows users to annotate workflow data dependencies with synchronization semantics; at run time, a user-space software layer automatically turns a batch execution into a streaming execution according to the semantics expressed in the configuration file.
CAPIO is a libre software available on Github (https://github.com/High-Performance-IO/capio) under the MIT license
Publications
2023
Alberto Riccardo Martinelli, Massimo Torquati, Marco Aldinucci, Iacopo Colonnelli, Barbara Cantalupo
CAPIO: a Middleware for Transparent I/O Streaming in Data-Intensive Workflows Proceedings Article
In: 2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC), IEEE, Goa, India, 2023.
Abstract | Links | BibTeX | Tags: admire, capio, eupex, icsc
@inproceedings{23:hipc:capio,
title = {CAPIO: a Middleware for Transparent I/O Streaming in Data-Intensive Workflows},
author = {Alberto Riccardo Martinelli and Massimo Torquati and Marco Aldinucci and Iacopo Colonnelli and Barbara Cantalupo},
url = {https://iris.unito.it/retrieve/27380f37-0978-409e-a9d8-2b5e95a4bb85/CAPIO-HiPC23-preprint.pdf},
doi = {10.1109/HiPC58850.2023.00031},
year = {2023},
date = {2023-12-01},
booktitle = {2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)},
publisher = {IEEE},
address = {Goa, India},
abstract = {With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to quickly expand, further exacerbating the performance gap between computing, memory, and storage technologies. This paper introduces CAPIO (Cross-Application Programmable I/O), a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. The contribution is twofold: 1) at design time, a new I/O coordination language allows users to annotate workflow data dependencies with synchronization semantics; 2) at run time, a user-space middleware automatically and transparently to the user turns a workflow batch execution into a streaming execution according to the semantics expressed in the configuration file. CAPIO has been tested on synthetic benchmarks simulating typical workflow I/O patterns and two real-world workflows. Experiments show that CAPIO reduces the execution time by 10% to 66% for data-intensive workflows that use the file system as a communication medium.},
keywords = {admire, capio, eupex, icsc},
pubstate = {published},
tppubtype = {inproceedings}
}
With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to quickly expand, further exacerbating the performance gap between computing, memory, and storage technologies. This paper introduces CAPIO (Cross-Application Programmable I/O), a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. The contribution is twofold: 1) at design time, a new I/O coordination language allows users to annotate workflow data dependencies with synchronization semantics; 2) at run time, a user-space middleware automatically and transparently to the user turns a workflow batch execution into a streaming execution according to the semantics expressed in the configuration file. CAPIO has been tested on synthetic benchmarks simulating typical workflow I/O patterns and two real-world workflows. Experiments show that CAPIO reduces the execution time by 10% to 66% for data-intensive workflows that use the file system as a communication medium.
Talks
2024
Marco Edoardo Santimaria, Iacopo Colonnelli, Marco Aldinucci
Releasing the CAPIO middleware from MPI derived constraints Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: across, admire, capio, capiocl, eupex, icsc
@misc{24:santimaria:bighpc,
title = {Releasing the CAPIO middleware from MPI derived constraints},
author = {Marco Edoardo Santimaria and Iacopo Colonnelli and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/zrJGD4i36fWdp5g},
year = {2024},
date = {2024-09-01},
address = {Pisa, Italy},
abstract = {CAPIO is a middleware that transparently injects streaming capabilities into file-based workflows. However, its implementation is limited to HPC environments based on the MPI framework, significantly limiting its applications. This paper will illustrate a proposed architecture and some preliminary results aimed at investigating the usage of a distributed files system as a communication media for the CAPIO middleware, with the ultimate goal of supporting both CLOUD-based and HPC-based workflows.},
keywords = {across, admire, capio, capiocl, eupex, icsc},
pubstate = {published},
tppubtype = {misc}
}
CAPIO is a middleware that transparently injects streaming capabilities into file-based workflows. However, its implementation is limited to HPC environments based on the MPI framework, significantly limiting its applications. This paper will illustrate a proposed architecture and some preliminary results aimed at investigating the usage of a distributed files system as a communication media for the CAPIO middleware, with the ultimate goal of supporting both CLOUD-based and HPC-based workflows.
Marco Edoardo Santimaria, Iacopo Colonnelli, Massimo Torquati, Marco Aldinucci
CAPIO: Cross Application Programamble IO Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: across, admire, capio, capiocl, eupex, icsc
@misc{24:santimaria:itadata:shpcpee,
title = {CAPIO: Cross Application Programamble IO},
author = {Marco Edoardo Santimaria and Iacopo Colonnelli and Massimo Torquati and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/rg6LWwrZXi6tTXm},
year = {2024},
date = {2024-09-01},
address = {Pisa, Italy},
abstract = {With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to expand, further exacerbating quickly the performance gap between computing, memory, and storage technologies. CAPIO (Cross-Application Programmable I/O), is a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. In this presentation, we will introduce the CAPIO-CL language with its semantics, as well as the implementation of the CAPIO-CL language through the CAPIO middleware. We will also provide some case studies of how CAPIO has been employed to improve workflow execution time as well as some future directions.},
keywords = {across, admire, capio, capiocl, eupex, icsc},
pubstate = {published},
tppubtype = {misc}
}
With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to expand, further exacerbating quickly the performance gap between computing, memory, and storage technologies. CAPIO (Cross-Application Programmable I/O), is a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation-I/O overlap without the need to change the application code. In this presentation, we will introduce the CAPIO-CL language with its semantics, as well as the implementation of the CAPIO-CL language through the CAPIO middleware. We will also provide some case studies of how CAPIO has been employed to improve workflow execution time as well as some future directions.
Marco Edoardo Santimaria
CAPIO-CL: Cross Application Programmable IO - Coordination Language Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: across, admire, capio, eupex, icsc
@misc{24:santimaria:hlpp:capiocl,
title = {CAPIO-CL: Cross Application Programmable IO - Coordination Language},
author = {Marco Edoardo Santimaria},
url = {https://datacloud.di.unito.it/index.php/s/zsKY3PWzX5NFCiX},
year = {2024},
date = {2024-07-01},
address = {Pisa, Italy},
abstract = {The performance bottleneck in file-based workflows remains a pressing issue in the realm of I/O-based workflows. To address this challenge, a novel annotation language has been developed. CAPIO-CL is positioned as an innovative I/O coordination language, enabling users to annotate data dependencies within file-based workflows with synchronization semantics pertinent to the involved files and directories. Through the information provided by the language, optimization opportunities arise in streaming and preemptive data movement. This paper serves to illustrate the semantics and syntax enabling CAPIO-CL to enhance the performance of in situ workflows without necessitating the rewriting or modification of the original workflow application steps. Finally, an analysis of CAPIO-CL is provided, taking into consideration both language expressiveness and application performance enhancement.},
keywords = {across, admire, capio, eupex, icsc},
pubstate = {published},
tppubtype = {misc}
}
The performance bottleneck in file-based workflows remains a pressing issue in the realm of I/O-based workflows. To address this challenge, a novel annotation language has been developed. CAPIO-CL is positioned as an innovative I/O coordination language, enabling users to annotate data dependencies within file-based workflows with synchronization semantics pertinent to the involved files and directories. Through the information provided by the language, optimization opportunities arise in streaming and preemptive data movement. This paper serves to illustrate the semantics and syntax enabling CAPIO-CL to enhance the performance of in situ workflows without necessitating the rewriting or modification of the original workflow application steps. Finally, an analysis of CAPIO-CL is provided, taking into consideration both language expressiveness and application performance enhancement.
FastFlow
A joint project with CS Dept. – University of Pisa
FastFlow (斋戒流) is a C++ parallel programming framework advocating high-level, pattern-based parallel programming. It chiefly supports streaming and data parallelism, targeting heterogenous platforms composed of clusters of shared-memory platforms, possibly equipped with computing accelerators such as NVidia GPGPUs, Xeon Phi, Tilera TILE64.
At today, FastFlow has been the background technology of 3 European Projects and 1 National project for an aggregate total cost of 12M € (ParaPhrase FP7, REPARA FP7, Rephrase H2020, and IMPACT, see projects section). We are still actively developing FastFlow along with its underlying technology, and we are wide open to turn challenges in research and innovation. More details can be found in the main FastFlow website.
FastFlow comes as a C++ template library designed as a stack of layers that progressively abstracts the programming of parallel applications. The goal of the stack is threefold: portability, extensibility, and performance. For this, all three layers are realized as thin strata of C++ templates that are 1) seamlessly portable, 2) easily extended via subclassing, and 3) statically compiled and cross-optimized with the application. The terse design ensures easy portability on almost all OSes and CPUs with a C++ compiler.
More details in the FastFlow website.
Publications
2020
Jose Daniel Garcia, Jose Daniel Rio, Marco Aldinucci, Fabio Tordini, Marco Danelutto, Gabriele Mencagli, Massimo Torquati
Challenging the abstraction penalty in parallel patterns libraries: Adding FastFlow support to GrPPI Journal Article
In: The Journal of Supercomputing, vol. 76, no. 7, pp. 5139–5159, 2020.
Abstract | Links | BibTeX | Tags: fastflow, rephrase
@article{19:jsupe:grppi,
title = {Challenging the abstraction penalty in parallel patterns libraries: Adding FastFlow support to GrPPI},
author = {Jose Daniel Garcia and Jose Daniel Rio and Marco Aldinucci and Fabio Tordini and Marco Danelutto and Gabriele Mencagli and Massimo Torquati},
url = {https://iris.unito.it/retrieve/handle/2318/1762686/744894/2020-js-grppi-postprint.pdf},
doi = {10.1007/s11227-019-02826-5},
year = {2020},
date = {2020-01-01},
journal = {The Journal of Supercomputing},
volume = {76},
number = {7},
pages = {5139–5159},
abstract = {In the last years, pattern-based programming has been recognized as a good practice for efficiently exploiting parallel hardware resources. Following this approach, multiple libraries have been designed for providing such high-level abstractions to ease the parallel programming. However, those libraries do not share a common interface. To pave the way, GrPPI has been designed for providing an intermediate abstraction layer between application developers and existing parallel programming frameworks like OpenMP, Intel TBB or ISO C++ threads. On the other hand, FastFlow has been adopted as an efficient object-based programming framework that may benefit from being supported as an additional GrPPI backend. However, the object-based approach presents some major challenges to be incorporated under the GrPPI type safe functional programming style. In this paper, we present the integration of FastFlow as a new GrPPI backend to demonstrate that structured parallel programming frameworks perfectly fit the GrPPI design. Additionally, we also demonstrate that GrPPI does not incur in additional overheads for providing its abstraction layer, and we study the programmability in terms of lines of code and cyclomatic complexity. In general, the presented work acts as reciprocal validation of both FastFlow (as an efficient, native structured parallel programming framework) and GrPPI (as an efficient abstraction layer on top of existing parallel programming frameworks).},
keywords = {fastflow, rephrase},
pubstate = {published},
tppubtype = {article}
}
In the last years, pattern-based programming has been recognized as a good practice for efficiently exploiting parallel hardware resources. Following this approach, multiple libraries have been designed for providing such high-level abstractions to ease the parallel programming. However, those libraries do not share a common interface. To pave the way, GrPPI has been designed for providing an intermediate abstraction layer between application developers and existing parallel programming frameworks like OpenMP, Intel TBB or ISO C++ threads. On the other hand, FastFlow has been adopted as an efficient object-based programming framework that may benefit from being supported as an additional GrPPI backend. However, the object-based approach presents some major challenges to be incorporated under the GrPPI type safe functional programming style. In this paper, we present the integration of FastFlow as a new GrPPI backend to demonstrate that structured parallel programming frameworks perfectly fit the GrPPI design. Additionally, we also demonstrate that GrPPI does not incur in additional overheads for providing its abstraction layer, and we study the programmability in terms of lines of code and cyclomatic complexity. In general, the presented work acts as reciprocal validation of both FastFlow (as an efficient, native structured parallel programming framework) and GrPPI (as an efficient abstraction layer on top of existing parallel programming frameworks).
2019
Marco Danelutto, Tiziano De Matteis, Daniele De Sensi, Gabriele Mencagli, Massimo Torquati, Marco Aldinucci, Peter Kilpatrick
The RePhrase Extended Pattern Set for Data Intensive Parallel Computing Journal Article
In: International Journal of Parallel Programming, vol. 47, no. 1, pp. 74–93, 2019.
Abstract | Links | BibTeX | Tags: fastflow, rephrase
@article{17:rephrasepatterns:ijpp,
title = {The RePhrase Extended Pattern Set for Data Intensive Parallel Computing},
author = {Marco Danelutto and Tiziano De Matteis and Daniele De Sensi and Gabriele Mencagli and Massimo Torquati and Marco Aldinucci and Peter Kilpatrick},
url = {https://iris.unito.it/retrieve/handle/2318/1659336/387667/2017_ijpp_rephrase.pdf},
doi = {10.1007/s10766-017-0540-z},
year = {2019},
date = {2019-01-01},
journal = {International Journal of Parallel Programming},
volume = {47},
number = {1},
pages = {74–93},
abstract = {We discuss the extended parallel pattern set identified within the EU-funded project RePhrase as a candidate pattern set to support data intensive applications targeting heterogeneous architectures. The set has been designed to include three classes of pattern, namely i) core patterns, modelling common, not necessarily data intensive parallelism exploitation patterns, usually to be used in composition; ii) high level patterns, modelling common, complex and complete parallelism exploitation patterns; and iii) building block patterns, modelling the single components of data intensive applications, suitable for use–in composition–to implement patterns not covered by the core and high level patterns. We discuss the expressive power of the RePhrase extended pattern set and results illustrating the performances that may be achieved with the FastFlow implementation of the high level patterns.},
keywords = {fastflow, rephrase},
pubstate = {published},
tppubtype = {article}
}
We discuss the extended parallel pattern set identified within the EU-funded project RePhrase as a candidate pattern set to support data intensive applications targeting heterogeneous architectures. The set has been designed to include three classes of pattern, namely i) core patterns, modelling common, not necessarily data intensive parallelism exploitation patterns, usually to be used in composition; ii) high level patterns, modelling common, complex and complete parallelism exploitation patterns; and iii) building block patterns, modelling the single components of data intensive applications, suitable for use–in composition–to implement patterns not covered by the core and high level patterns. We discuss the expressive power of the RePhrase extended pattern set and results illustrating the performances that may be achieved with the FastFlow implementation of the high level patterns.
Massimo Torquati, Gabriele Mencagli, Maurizio Drocco, Marco Aldinucci, Tiziano De Matteis, Marco Danelutto
On Dynamic Memory Allocation in Sliding-Window Parallel Patterns for Streaming Analytics Journal Article
In: The Journal of Supercomputing, vol. 75, no. 8, pp. 4114–4131, 2019.
Abstract | Links | BibTeX | Tags: fastflow, rephrase
@article{17:dmadasp:jsupe,
title = {On Dynamic Memory Allocation in Sliding-Window Parallel Patterns for Streaming Analytics},
author = {Massimo Torquati and Gabriele Mencagli and Maurizio Drocco and Marco Aldinucci and Tiziano De Matteis and Marco Danelutto},
url = {https://iris.unito.it/retrieve/handle/2318/1648626/362381/17_torquati_jsc.pdf},
doi = {10.1007/s11227-017-2152-1},
year = {2019},
date = {2019-01-01},
journal = {The Journal of Supercomputing},
volume = {75},
number = {8},
pages = {4114–4131},
abstract = {This work studies the issues related to dynamic memory management in Data Stream Processing, an emerging paradigm enabling the real-time processing of live data streams. In this paper we consider two streaming parallel patterns and we discuss different implementation variants related on how dynamic memory is managed. The results show that the standard mechanisms provided by modern C++ are not entirely adequate for maximizing the performance. Instead, the combined use of an efficient general-purpose memory allocator, a custom allocator optimized for the pattern considered and a custom variant of the C++ shared pointer mechanism, provides a performance improvement up to 16% on the best case.},
keywords = {fastflow, rephrase},
pubstate = {published},
tppubtype = {article}
}
This work studies the issues related to dynamic memory management in Data Stream Processing, an emerging paradigm enabling the real-time processing of live data streams. In this paper we consider two streaming parallel patterns and we discuss different implementation variants related on how dynamic memory is managed. The results show that the standard mechanisms provided by modern C++ are not entirely adequate for maximizing the performance. Instead, the combined use of an efficient general-purpose memory allocator, a custom allocator optimized for the pattern considered and a custom variant of the C++ shared pointer mechanism, provides a performance improvement up to 16% on the best case.
2018
Gabriele Mencagli, Massimo Torquati, Fabio Lucattini, Salvatore Cuomo, Marco Aldinucci
Harnessing sliding-window execution semantics for parallel stream processing Journal Article
In: Journal of Parallel and Distributed Computing, vol. 116, pp. 74–88, 2018, ISSN: 0743-7315.
Abstract | Links | BibTeX | Tags: fastflow, rephrase
@article{17:slidingwindows:jpdc,
title = {Harnessing sliding-window execution semantics for parallel stream processing},
author = {Gabriele Mencagli and Massimo Torquati and Fabio Lucattini and Salvatore Cuomo and Marco Aldinucci},
url = {https://iris.unito.it/retrieve/e27ce42c-1381-2581-e053-d805fe0acbaa/preprint-jpdc-2017.pdf},
doi = {10.1016/j.jpdc.2017.10.021},
issn = {0743-7315},
year = {2018},
date = {2018-06-01},
journal = {Journal of Parallel and Distributed Computing},
volume = {116},
pages = {74–88},
abstract = {Abstract According to the recent trend in data acquisition and processing technology, big data are increasingly available in the form of unbounded streams of elementary data items to be processed in real-time. In this paper we study in detail the paradigm of sliding windows, a well-known technique for approximated queries that update their results continuously as new fresh data arrive from the stream. In this work we focus on the relationship between the various existing sliding window semantics and the way the query processing is performed from the parallelism perspective. From this study two alternative parallel models are identified, each covering semantics with very precise properties. Each model is described in terms of its pros and cons, and parallel implementations in the FastFlow framework are analyzed by discussing the layout of the concurrent data structures used for the efficient windows representation in each model.},
keywords = {fastflow, rephrase},
pubstate = {published},
tppubtype = {article}
}
Abstract According to the recent trend in data acquisition and processing technology, big data are increasingly available in the form of unbounded streams of elementary data items to be processed in real-time. In this paper we study in detail the paradigm of sliding windows, a well-known technique for approximated queries that update their results continuously as new fresh data arrive from the stream. In this work we focus on the relationship between the various existing sliding window semantics and the way the query processing is performed from the parallelism perspective. From this study two alternative parallel models are identified, each covering semantics with very precise properties. Each model is described in terms of its pros and cons, and parallel implementations in the FastFlow framework are analyzed by discussing the layout of the concurrent data structures used for the efficient windows representation in each model.
Claudia Misale, Maurizio Drocco, Guy Tremblay, Alberto R. Martinelli, Marco Aldinucci
PiCo: High-performance data analytics pipelines in modern C++ Journal Article
In: Future Generation Computer Systems, vol. 87, pp. 392–403, 2018.
Abstract | Links | BibTeX | Tags: fastflow, HPC, toreador
@article{18:fgcs:pico,
title = {PiCo: High-performance data analytics pipelines in modern C++},
author = {Claudia Misale and Maurizio Drocco and Guy Tremblay and Alberto R. Martinelli and Marco Aldinucci},
url = {https://iris.unito.it/retrieve/handle/2318/1668444/414280/fgcs_pico.pdf},
doi = {10.1016/j.future.2018.05.030},
year = {2018},
date = {2018-01-01},
booktitle = {Future Generation Computer Systems},
journal = {Future Generation Computer Systems},
volume = {87},
pages = {392–403},
abstract = {In this paper, we present a new C++ API with a fluent interface called PiCo (Pipeline Composition). PiCo's programming model aims at making easier the programming of data analytics applications while preserving or enhancing their performance. This is attained through three key design choices: (1) unifying batch and stream data access models, (2) decoupling processing from data layout, and (3) exploiting a stream-oriented, scalable, efficient C++11 runtime system. PiCo proposes a programming model based on pipelines and operators that are polymorphic with respect to data types in the sense that it is possible to reuse the same algorithms and pipelines on different data models (e.g., streams, lists, sets, etc.). Preliminary results show that PiCo, when compared to Spark and Flink, can attain better performances in terms of execution times and can hugely improve memory utilization, both for batch and stream processing.},
keywords = {fastflow, HPC, toreador},
pubstate = {published},
tppubtype = {article}
}
In this paper, we present a new C++ API with a fluent interface called PiCo (Pipeline Composition). PiCo's programming model aims at making easier the programming of data analytics applications while preserving or enhancing their performance. This is attained through three key design choices: (1) unifying batch and stream data access models, (2) decoupling processing from data layout, and (3) exploiting a stream-oriented, scalable, efficient C++11 runtime system. PiCo proposes a programming model based on pipelines and operators that are polymorphic with respect to data types in the sense that it is possible to reuse the same algorithms and pipelines on different data models (e.g., streams, lists, sets, etc.). Preliminary results show that PiCo, when compared to Spark and Flink, can attain better performances in terms of execution times and can hugely improve memory utilization, both for batch and stream processing.
2017
Maurizio Drocco
Parallel Programming with Global Asynchronous Memory: Models, C++ APIs and Implementations PhD Thesis
Computer Science Department, University of Torino, 2017.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase, repara, rephrase, toreador
@phdthesis{17:gam:drocco:thesis,
title = {Parallel Programming with Global Asynchronous Memory: Models, C++ APIs and Implementations},
author = {Maurizio Drocco},
url = {https://zenodo.org/record/1037585/files/Drocco_phd_thesis.pdf},
doi = {10.5281/zenodo.1037585},
year = {2017},
date = {2017-10-01},
school = {Computer Science Department, University of Torino},
abstract = {In the realm of High Performance Computing (HPC), message passing has been the programming paradigm of choice for over twenty years. The durable MPI (Message Passing Interface) standard, with send/receive communication, broadcast, gather/scatter, and reduction collectives is still used to construct parallel programs where each communication is orchestrated by the de-vel-oper-based precise knowledge of data distribution and overheads; collective communications simplify the orchestration but might induce excessive synchronization. Early attempts to bring shared-memory programming model—with its programming adv-antages—to distributed computing, referred as the Distributed Shared Memory (DSM) model, faded away; one of the main issue was to combine performance and programmability with the memory consistency model. The recently proposed Partitioned Global Address Space (PGAS) model is a modern revamp of DSM that exposes data placement to enable optimizations based on locality, but it still addresses (simple) data-parallelism only and it relies on expensive sharing protocols. We advocate an alternative programming model for distributed computing based on a Global Asynchronous Memory (GAM), aiming to emphavoid coherency and consistency problems rather than solving them. We materialize GAM by designing and implementing a emphdistributed smart pointers library, inspired by C++ smart pointers. In this model, public and private pointers (resembling C++ shared and unique pointers, respectively) are moved around instead of messages (i.e., data), thus alleviating the user from the burden of minimizing transfers. On top of smart pointers, we propose a high-level C++ template library for writing applications in terms of dataflow-like networks, namely GAM nets, consisting of stateful processors exchanging pointers in fully asynchronous fashion. We demonstrate the validity of the proposed approach, from the expressiveness perspective, by showing how GAM nets can be exploited to implement higher-level parallel programming models, such as data and task parallelism. As for the performance perspective, the execution of two non-toy benchmarks on a number of different small-scale HPC clusters exhibits both close-to-ideal scalability and negligible overhead with respect to state-of-the-art benchmark implementations. For instance, the GAM implementation of a high-quality video restoration filter sustains a 100 fps throughput over 70%-noisy high-quality video streams on a 4-node cluster of Graphics Processing Units (GPUs), with minimal programming effort.},
keywords = {fastflow, paraphrase, repara, rephrase, toreador},
pubstate = {published},
tppubtype = {phdthesis}
}
In the realm of High Performance Computing (HPC), message passing has been the programming paradigm of choice for over twenty years. The durable MPI (Message Passing Interface) standard, with send/receive communication, broadcast, gather/scatter, and reduction collectives is still used to construct parallel programs where each communication is orchestrated by the de-vel-oper-based precise knowledge of data distribution and overheads; collective communications simplify the orchestration but might induce excessive synchronization. Early attempts to bring shared-memory programming model—with its programming adv-antages—to distributed computing, referred as the Distributed Shared Memory (DSM) model, faded away; one of the main issue was to combine performance and programmability with the memory consistency model. The recently proposed Partitioned Global Address Space (PGAS) model is a modern revamp of DSM that exposes data placement to enable optimizations based on locality, but it still addresses (simple) data-parallelism only and it relies on expensive sharing protocols. We advocate an alternative programming model for distributed computing based on a Global Asynchronous Memory (GAM), aiming to emphavoid coherency and consistency problems rather than solving them. We materialize GAM by designing and implementing a emphdistributed smart pointers library, inspired by C++ smart pointers. In this model, public and private pointers (resembling C++ shared and unique pointers, respectively) are moved around instead of messages (i.e., data), thus alleviating the user from the burden of minimizing transfers. On top of smart pointers, we propose a high-level C++ template library for writing applications in terms of dataflow-like networks, namely GAM nets, consisting of stateful processors exchanging pointers in fully asynchronous fashion. We demonstrate the validity of the proposed approach, from the expressiveness perspective, by showing how GAM nets can be exploited to implement higher-level parallel programming models, such as data and task parallelism. As for the performance perspective, the execution of two non-toy benchmarks on a number of different small-scale HPC clusters exhibits both close-to-ideal scalability and negligible overhead with respect to state-of-the-art benchmark implementations. For instance, the GAM implementation of a high-quality video restoration filter sustains a 100 fps throughput over 70%-noisy high-quality video streams on a 4-node cluster of Graphics Processing Units (GPUs), with minimal programming effort.
Claudia Misale
PiCo: A Domain-Specific Language for Data Analytics Pipelines PhD Thesis
Computer Science Department, University of Torino, 2017.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase, repara, rephrase, toreador
@phdthesis{17:pico:misale:thesis,
title = {PiCo: A Domain-Specific Language for Data Analytics Pipelines},
author = {Claudia Misale},
url = {https://iris.unito.it/retrieve/handle/2318/1633743/320170/Misale_thesis.pdf},
doi = {10.5281/zenodo.579753},
year = {2017},
date = {2017-05-01},
school = {Computer Science Department, University of Torino},
abstract = {In the world of Big Data analytics, there is a series of tools aiming at simplifying programming applications to be executed on clusters. Although each tool claims to provide better programming, data and execution models—for which only informal (and often confusing) semantics is generally provided—all share a common under- lying model, namely, the Dataflow model. Using this model as a starting point, it is possible to categorize and analyze almost all aspects about Big Data analytics tools from a high level perspective. This analysis can be considered as a first step toward a formal model to be exploited in the design of a (new) framework for Big Data analytics. By putting clear separations between all levels of abstraction (i.e., from the runtime to the user API), it is easier for a programmer or software designer to avoid mixing low level with high level aspects, as we are often used to see in state-of-the-art Big Data analytics frameworks.
From the user-level perspective, we think that a clearer and simple semantics is preferable, together with a strong separation of concerns. For this reason, we use the Dataflow model as a starting point to build a programming environment with a simplified programming model implemented as a Domain-Specific Language, that is on top of a stack of layers that build a prototypical framework for Big Data analytics.
The contribution of this thesis is twofold: first, we show that the proposed model is (at least) as general as existing batch and streaming frameworks (e.g., Spark, Flink, Storm, Google Dataflow), thus making it easier to understand high-level data-processing applications written in such frameworks. As result of this analysis, we provide a layered model that can represent tools and applications following the Dataflow paradigm and we show how the analyzed tools fit in each level.
Second, we propose a programming environment based on such layered model in the form of a Domain-Specific Language (DSL) for processing data collections, called PiCo (Pipeline Composition). The main entity of this programming model is the Pipeline, basically a DAG-composition of processing elements. This model is intended to give the user an unique interface for both stream and batch processing, hiding completely data management and focusing only on operations, which are represented by Pipeline stages. Our DSL will be built on top of the FastFlow library, exploiting both shared and distributed parallelism, and implemented in C++11/14 with the aim of porting C++ into the Big Data world.},
keywords = {fastflow, paraphrase, repara, rephrase, toreador},
pubstate = {published},
tppubtype = {phdthesis}
}
In the world of Big Data analytics, there is a series of tools aiming at simplifying programming applications to be executed on clusters. Although each tool claims to provide better programming, data and execution models—for which only informal (and often confusing) semantics is generally provided—all share a common under- lying model, namely, the Dataflow model. Using this model as a starting point, it is possible to categorize and analyze almost all aspects about Big Data analytics tools from a high level perspective. This analysis can be considered as a first step toward a formal model to be exploited in the design of a (new) framework for Big Data analytics. By putting clear separations between all levels of abstraction (i.e., from the runtime to the user API), it is easier for a programmer or software designer to avoid mixing low level with high level aspects, as we are often used to see in state-of-the-art Big Data analytics frameworks.
From the user-level perspective, we think that a clearer and simple semantics is preferable, together with a strong separation of concerns. For this reason, we use the Dataflow model as a starting point to build a programming environment with a simplified programming model implemented as a Domain-Specific Language, that is on top of a stack of layers that build a prototypical framework for Big Data analytics.
The contribution of this thesis is twofold: first, we show that the proposed model is (at least) as general as existing batch and streaming frameworks (e.g., Spark, Flink, Storm, Google Dataflow), thus making it easier to understand high-level data-processing applications written in such frameworks. As result of this analysis, we provide a layered model that can represent tools and applications following the Dataflow paradigm and we show how the analyzed tools fit in each level.
Second, we propose a programming environment based on such layered model in the form of a Domain-Specific Language (DSL) for processing data collections, called PiCo (Pipeline Composition). The main entity of this programming model is the Pipeline, basically a DAG-composition of processing elements. This model is intended to give the user an unique interface for both stream and batch processing, hiding completely data management and focusing only on operations, which are represented by Pipeline stages. Our DSL will be built on top of the FastFlow library, exploiting both shared and distributed parallelism, and implemented in C++11/14 with the aim of porting C++ into the Big Data world.
From the user-level perspective, we think that a clearer and simple semantics is preferable, together with a strong separation of concerns. For this reason, we use the Dataflow model as a starting point to build a programming environment with a simplified programming model implemented as a Domain-Specific Language, that is on top of a stack of layers that build a prototypical framework for Big Data analytics.
The contribution of this thesis is twofold: first, we show that the proposed model is (at least) as general as existing batch and streaming frameworks (e.g., Spark, Flink, Storm, Google Dataflow), thus making it easier to understand high-level data-processing applications written in such frameworks. As result of this analysis, we provide a layered model that can represent tools and applications following the Dataflow paradigm and we show how the analyzed tools fit in each level.
Second, we propose a programming environment based on such layered model in the form of a Domain-Specific Language (DSL) for processing data collections, called PiCo (Pipeline Composition). The main entity of this programming model is the Pipeline, basically a DAG-composition of processing elements. This model is intended to give the user an unique interface for both stream and batch processing, hiding completely data management and focusing only on operations, which are represented by Pipeline stages. Our DSL will be built on top of the FastFlow library, exploiting both shared and distributed parallelism, and implemented in C++11/14 with the aim of porting C++ into the Big Data world.
Marco Aldinucci, Marco Danelutto, Daniele De Sensi, Gabriele Mencagli, Massimo Torquati
Towards Power-Aware Data Pipelining on Multicores Proceedings Article
In: Proceedings of the 10th International Symposium on High-Level Parallel Programming and Applications, Valladolid, Spain, 2017.
Abstract | Links | BibTeX | Tags: fastflow, rephrase
@inproceedings{17:hlpp:powerstream,
title = {Towards Power-Aware Data Pipelining on Multicores},
author = {Marco Aldinucci and Marco Danelutto and Daniele De Sensi and Gabriele Mencagli and Massimo Torquati},
url = {https://iris.unito.it/retrieve/handle/2318/1644982/351415/17_HLPP_powerstream.pdf},
year = {2017},
date = {2017-01-01},
booktitle = {Proceedings of the 10th International Symposium on High-Level Parallel Programming and Applications},
address = {Valladolid, Spain},
abstract = {Power consumption management has become a major concern in software development. Continuous streaming computations are usually com- posed by different modules, exchanging data through shared message queues. The selection of the algorithm used to access such queues (i.e., the concurrency control) is a critical aspect for both performance and power consumption. In this paper, we describe the design of an adaptive concurrency control algo- rithm for implementing power-efficient communications on shared memory multicores. The algorithm provides the throughput offered by a nonblocking implementation and the power efficiency of a blocking protocol. We demon- strate that our algorithm reduces the power consumption of data streaming computations without decreasing their throughput.},
keywords = {fastflow, rephrase},
pubstate = {published},
tppubtype = {inproceedings}
}
Power consumption management has become a major concern in software development. Continuous streaming computations are usually com- posed by different modules, exchanging data through shared message queues. The selection of the algorithm used to access such queues (i.e., the concurrency control) is a critical aspect for both performance and power consumption. In this paper, we describe the design of an adaptive concurrency control algo- rithm for implementing power-efficient communications on shared memory multicores. The algorithm provides the throughput offered by a nonblocking implementation and the power efficiency of a blocking protocol. We demon- strate that our algorithm reduces the power consumption of data streaming computations without decreasing their throughput.
Marco Aldinucci, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
FastFlow: high-level and efficient streaming on multi-core Book Chapter
In: Pllana, Sabri, Xhafa, Fatos (Ed.): Programming Multi-core and Many-core Computing Systems, Chapter 13, pp. 261–280, John Wiley & Sons, Ltd, 2017, ISBN: 9781119332015.
Abstract | Links | BibTeX | Tags: fastflow
@inbook{ff:wileybook:17,
title = {FastFlow: high-level and efficient streaming on multi-core},
author = {Marco Aldinucci and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
editor = {Sabri Pllana and Fatos Xhafa},
url = {http://calvados.di.unipi.it/storage/paper_files/2011_FF_tutorial-draft.pdf},
doi = {10.1002/9781119332015.ch13},
isbn = {9781119332015},
year = {2017},
date = {2017-01-01},
booktitle = {Programming Multi-core and Many-core Computing Systems},
pages = {261–280},
publisher = {John Wiley & Sons, Ltd},
chapter = {13},
series = {Parallel and Distributed Computing},
abstract = {This chapter first outlines FastFlow design and then shows sample use of the FastFlow programming environment together with performance results achieved on various state-of-the-art multicore architectures. The FastFlow framework has been designed according to four foundational principles: layered design; efficiency in base mechanisms; support for stream parallelism; and a programming model based on design pattern/algorithmic skeleton concepts. The core of the FastFlow framework provides an efficient implementation of single-producer-single-consumer (SPSC) first in-first out (FIFO) queues. The next tier up extends from one-to-one queues to one-to-many, many-to-one, and many-to-many synchronizations and data flows, which are implemented using only SPSC queues and arbiter threads, thus providing lock-free and wait-free arbitrary dataflow graphs. When designing and implementing new parallel applications using FastFlow, programmers instantiate patterns provided by FastFlow to adapt them to the specific needs of the application at hand. The chapter demonstrates how the principal FastFlow patterns may be used in a parallel application.},
keywords = {fastflow},
pubstate = {published},
tppubtype = {inbook}
}
This chapter first outlines FastFlow design and then shows sample use of the FastFlow programming environment together with performance results achieved on various state-of-the-art multicore architectures. The FastFlow framework has been designed according to four foundational principles: layered design; efficiency in base mechanisms; support for stream parallelism; and a programming model based on design pattern/algorithmic skeleton concepts. The core of the FastFlow framework provides an efficient implementation of single-producer-single-consumer (SPSC) first in-first out (FIFO) queues. The next tier up extends from one-to-one queues to one-to-many, many-to-one, and many-to-many synchronizations and data flows, which are implemented using only SPSC queues and arbiter threads, thus providing lock-free and wait-free arbitrary dataflow graphs. When designing and implementing new parallel applications using FastFlow, programmers instantiate patterns provided by FastFlow to adapt them to the specific needs of the application at hand. The chapter demonstrates how the principal FastFlow patterns may be used in a parallel application.
Fabio Tordini, Maurizio Drocco, Claudia Misale, Luciano Milanesi, Pietro Liò, Ivan Merelli, Massimo Torquati, Marco Aldinucci
NuChart-II: the road to a fast and scalable tool for Hi-C data analysis Journal Article
In: International Journal of High Performance Computing Applications, vol. 31, no. 3, pp. 196–211, 2017.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, repara, rephrase
@article{16:ijhpca:nuchart,
title = {NuChart-II: the road to a fast and scalable tool for Hi-C data analysis},
author = {Fabio Tordini and Maurizio Drocco and Claudia Misale and Luciano Milanesi and Pietro Liò and Ivan Merelli and Massimo Torquati and Marco Aldinucci},
url = {https://iris.unito.it/retrieve/handle/2318/1607126/238747/main.pdf},
doi = {10.1177/1094342016668567},
year = {2017},
date = {2017-01-01},
journal = {International Journal of High Performance Computing Applications},
volume = {31},
number = {3},
pages = {196–211},
abstract = {Recent advances in molecular biology and bioinformatics techniques brought to an explosion of the information about the spatial organisation of the DNA in the nucleus of a cell. High-throughput molecular biology techniques provide a genome-wide capture of the spatial organization of chromosomes at unprecedented scales, which permit to identify physical interactions between genetic elements located throughout a genome. Recent results have shown that there is a large correlation between co-localization and co-regulation of genes, but these important information are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we present NuChart-II, an efficient and highly optimized tool for genomic data analysis that provides a gene-centric, graph-based representation of genomic information. While designing NuChart-II we addressed several common issues in the parallelisation of memory bound algorithms for shared-memory systems. With performance and usability in mind, NuChart-II is a R package that embeds a C++ engine: computing capabilities and memory hierarchy of multi-core architectures are fully exploited, while the versatile R environment for statistical analysis and data visualisation rises the level of abstraction and permits to orchestrate analysis and visualisation of genomic data.},
keywords = {bioinformatics, fastflow, repara, rephrase},
pubstate = {published},
tppubtype = {article}
}
Recent advances in molecular biology and bioinformatics techniques brought to an explosion of the information about the spatial organisation of the DNA in the nucleus of a cell. High-throughput molecular biology techniques provide a genome-wide capture of the spatial organization of chromosomes at unprecedented scales, which permit to identify physical interactions between genetic elements located throughout a genome. Recent results have shown that there is a large correlation between co-localization and co-regulation of genes, but these important information are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we present NuChart-II, an efficient and highly optimized tool for genomic data analysis that provides a gene-centric, graph-based representation of genomic information. While designing NuChart-II we addressed several common issues in the parallelisation of memory bound algorithms for shared-memory systems. With performance and usability in mind, NuChart-II is a R package that embeds a C++ engine: computing capabilities and memory hierarchy of multi-core architectures are fully exploited, while the versatile R environment for statistical analysis and data visualisation rises the level of abstraction and permits to orchestrate analysis and visualisation of genomic data.
2016
Fabio Tordini
The road towards a Cloud-based High-Performance solution for genomic data analysis PhD Thesis
Computer Science Department, University of Torino, Italy, 2016.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow
@phdthesis{tordiniThesis16,
title = {The road towards a Cloud-based High-Performance solution for genomic data analysis},
author = {Fabio Tordini},
url = {http://calvados.di.unipi.it/storage/paper_files/2016_tordini_phdthesis.pdf},
year = {2016},
date = {2016-04-01},
school = {Computer Science Department, University of Torino, Italy},
abstract = {Nowadays, molecular biology laboratories are delivering more and more data about DNA organisation, at increasing resolution and in a large number of samples. So much that genomic research is now facing many of the scale-out issues that high-performance computing has been addressing for years: they require powerful infrastructures with fast computing and storage capabilities, with substantial challenges in terms of data processing, statistical analysis and data representation. With this thesis we propose a high-performance pipeline for the analysis and interpretation of heterogeneous genomic information: beside performance, usability and availability are two essential requirements that novel Bioinformatics tools should satisfy. In this perspective, we propose and discuss our efforts towards a solid infrastructure for data processing and storage, where software that operates over data is exposed as a service, and is accessible by users through the Internet. We begin by presenting NuChart-II, a tool for the analysis and interpretation of spatial genomic information. With NuChart-II we propose a graph-based representation of genomic data, which can provide insights on the disposition of genomic elements in the DNA. We also discuss our approach for the normalisation of biases that affect raw sequenced data. We believe that many currently available tools for genomic data analysis are perceived as tricky and troublesome applications, that require highly specialised skills to obtain the desired outcomes. Concerning usability, we want to rise the level of abstraction perceived by the user, but maintain high performance and correctness while providing an exhaustive solution for data visualisation. We also intend to foster the availability of novel tools: in this work we also discuss a cloud solution that delivers computation and storage as dynamically allocated virtual resources via the Internet, while needed software is provided as a service. In this way, the computational demand of genomic research can be satisfied more economically by using lab-scale and enterprise-oriented technologies. Here we discuss our idea of a task farm for the integration of heterogeneous data resulting from different sequencing experiments: we believe that the integration of multi-omic features on a nuclear map can be a valuable mean for studying the interactions among genetic elements. This can reveal insights on biological mechanisms, such as genes regulation, translocations and epigenetic patterns.},
keywords = {bioinformatics, fastflow},
pubstate = {published},
tppubtype = {phdthesis}
}
Nowadays, molecular biology laboratories are delivering more and more data about DNA organisation, at increasing resolution and in a large number of samples. So much that genomic research is now facing many of the scale-out issues that high-performance computing has been addressing for years: they require powerful infrastructures with fast computing and storage capabilities, with substantial challenges in terms of data processing, statistical analysis and data representation. With this thesis we propose a high-performance pipeline for the analysis and interpretation of heterogeneous genomic information: beside performance, usability and availability are two essential requirements that novel Bioinformatics tools should satisfy. In this perspective, we propose and discuss our efforts towards a solid infrastructure for data processing and storage, where software that operates over data is exposed as a service, and is accessible by users through the Internet. We begin by presenting NuChart-II, a tool for the analysis and interpretation of spatial genomic information. With NuChart-II we propose a graph-based representation of genomic data, which can provide insights on the disposition of genomic elements in the DNA. We also discuss our approach for the normalisation of biases that affect raw sequenced data. We believe that many currently available tools for genomic data analysis are perceived as tricky and troublesome applications, that require highly specialised skills to obtain the desired outcomes. Concerning usability, we want to rise the level of abstraction perceived by the user, but maintain high performance and correctness while providing an exhaustive solution for data visualisation. We also intend to foster the availability of novel tools: in this work we also discuss a cloud solution that delivers computation and storage as dynamically allocated virtual resources via the Internet, while needed software is provided as a service. In this way, the computational demand of genomic research can be satisfied more economically by using lab-scale and enterprise-oriented technologies. Here we discuss our idea of a task farm for the integration of heterogeneous data resulting from different sequencing experiments: we believe that the integration of multi-omic features on a nuclear map can be a valuable mean for studying the interactions among genetic elements. This can reveal insights on biological mechanisms, such as genes regulation, translocations and epigenetic patterns.
Maurizio Drocco, Claudia Misale, Marco Aldinucci
A Cluster-As-Accelerator approach for SPMD-free Data Parallelism Proceedings Article
In: Proc. of 24th Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP), pp. 350–353, IEEE, Crete, Greece, 2016.
Abstract | Links | BibTeX | Tags: fastflow, rephrase
@inproceedings{skedato:pdp:16,
title = {A Cluster-As-Accelerator approach for SPMD-free Data Parallelism},
author = {Maurizio Drocco and Claudia Misale and Marco Aldinucci},
url = {https://iris.unito.it/retrieve/handle/2318/1611858/262689/2016_pdp_skedato.pdf},
doi = {10.1109/PDP.2016.97},
year = {2016},
date = {2016-01-01},
booktitle = {Proc. of 24th Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)},
pages = {350–353},
publisher = {IEEE},
address = {Crete, Greece},
abstract = {In this paper we present a novel approach for functional-style programming of distributed-memory clusters, targeting data-centric applications. The programming model proposed is purely sequential, SPMD-free and based on high- level functional features introduced since C++11 specification. Additionally, we propose a novel cluster-as-accelerator design principle. In this scheme, cluster nodes act as general inter- preters of user-defined functional tasks over node-local portions of distributed data structures. We envision coupling a simple yet powerful programming model with a lightweight, locality- aware distributed runtime as a promising step along the road towards high-performance data analytics, in particular under the perspective of the upcoming exascale era. We implemented the proposed approach in SkeDaTo, a prototyping C++ library of data-parallel skeletons exploiting cluster-as-accelerator at the bottom layer of the runtime software stack.},
keywords = {fastflow, rephrase},
pubstate = {published},
tppubtype = {inproceedings}
}
In this paper we present a novel approach for functional-style programming of distributed-memory clusters, targeting data-centric applications. The programming model proposed is purely sequential, SPMD-free and based on high- level functional features introduced since C++11 specification. Additionally, we propose a novel cluster-as-accelerator design principle. In this scheme, cluster nodes act as general inter- preters of user-defined functional tasks over node-local portions of distributed data structures. We envision coupling a simple yet powerful programming model with a lightweight, locality- aware distributed runtime as a promising step along the road towards high-performance data analytics, in particular under the perspective of the upcoming exascale era. We implemented the proposed approach in SkeDaTo, a prototyping C++ library of data-parallel skeletons exploiting cluster-as-accelerator at the bottom layer of the runtime software stack.
Vladimir Janjic, Christopher Brown, Kenneth MacKenzie, Kevin Hammond, Marco Danelutto, Marco Aldinucci, Jose Daniel Garcia
RPL: A Domain-Specific Language for Designing and Implementing Parallel C++ Applications Proceedings Article
In: Proc. of Intl. Euromicro PDP 2016: Parallel Distributed and network-based Processing, IEEE, Crete, Greece, 2016.
Abstract | Links | BibTeX | Tags: fastflow, rephrase
@inproceedings{rpl:pdp:16,
title = {RPL: A Domain-Specific Language for Designing and Implementing Parallel C++ Applications},
author = {Vladimir Janjic and Christopher Brown and Kenneth MacKenzie and Kevin Hammond and Marco Danelutto and Marco Aldinucci and Jose Daniel Garcia},
url = {https://iris.unito.it/retrieve/handle/2318/1597172/299237/2016_jsupe_stencil_pp_4aperto.pdf},
doi = {10.1109/PDP.2016.122},
year = {2016},
date = {2016-01-01},
booktitle = {Proc. of Intl. Euromicro PDP 2016: Parallel Distributed and network-based Processing},
publisher = {IEEE},
address = {Crete, Greece},
abstract = {Parallelising sequential applications is usually a very hard job, due to many different ways in which an application can be parallelised and a large number of programming models (each with its own advantages and disadvantages) that can be used. In this paper, we describe a method to semi- automatically generate and evaluate different parallelisations of the same application, allowing programmers to find the best parallelisation without significant manual reengineering of the code. We describe a novel, high-level domain-specific language, Refactoring Pattern Language (RPL), that is used to represent the parallel structure of an application and to capture its extra-functional properties (such as service time). We then describe a set of RPL rewrite rules that can be used to generate alternative, but semantically equivalent, parallel structures (parallelisations) of the same application. We also describe the RPL Shell that can be used to evaluate these parallelisations, in terms of the desired extra-functional properties. Finally, we describe a set of C++ refactorings, targeting OpenMP, Intel TBB and FastFlow parallel programming models, that semi-automatically apply the desired parallelisation to the application's source code, therefore giving a parallel version of the code. We demonstrate how the RPL and the refactoring rules can be used to derive efficient parallelisations of two realistic C++ use cases (Image Convolution and Ant Colony Optimisation).},
keywords = {fastflow, rephrase},
pubstate = {published},
tppubtype = {inproceedings}
}
Parallelising sequential applications is usually a very hard job, due to many different ways in which an application can be parallelised and a large number of programming models (each with its own advantages and disadvantages) that can be used. In this paper, we describe a method to semi- automatically generate and evaluate different parallelisations of the same application, allowing programmers to find the best parallelisation without significant manual reengineering of the code. We describe a novel, high-level domain-specific language, Refactoring Pattern Language (RPL), that is used to represent the parallel structure of an application and to capture its extra-functional properties (such as service time). We then describe a set of RPL rewrite rules that can be used to generate alternative, but semantically equivalent, parallel structures (parallelisations) of the same application. We also describe the RPL Shell that can be used to evaluate these parallelisations, in terms of the desired extra-functional properties. Finally, we describe a set of C++ refactorings, targeting OpenMP, Intel TBB and FastFlow parallel programming models, that semi-automatically apply the desired parallelisation to the application's source code, therefore giving a parallel version of the code. We demonstrate how the RPL and the refactoring rules can be used to derive efficient parallelisations of two realistic C++ use cases (Image Convolution and Ant Colony Optimisation).
Marco Aldinucci, Sonia Campa, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
Pool Evolution: A Parallel Pattern for Evolutionary and Symbolic Computing Journal Article
In: International Journal of Parallel Programming, vol. 44, no. 3, pp. 531–551, 2016, ISSN: 0885-7458.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase, repara
@article{pool:ijpp:15,
title = {Pool Evolution: A Parallel Pattern for Evolutionary and Symbolic Computing},
author = {Marco Aldinucci and Sonia Campa and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
url = {https://iris.unito.it/retrieve/handle/2318/1522392/42139/2015_ff_pool_ijpp.pdf},
doi = {10.1007/s10766-015-0358-5},
issn = {0885-7458},
year = {2016},
date = {2016-01-01},
journal = {International Journal of Parallel Programming},
volume = {44},
number = {3},
pages = {531–551},
publisher = {Springer US},
abstract = {We introduce a new parallel pattern derived from a specific application domain and show how it turns out to have application beyond its domain of origin. The pool evolution pattern models the parallel evolution of a population subject to mutations and evolving in such a way that a given fitness function is optimized. The pattern has been demonstrated to be suitable for capturing and modeling the parallel patterns underpinning various evolutionary algorithms, as well as other parallel patterns typical of symbolic computation. In this paper we introduce the pattern, we discuss its implementation on modern multi/many core architectures and finally present experimental results obtained with FastFlow and Erlang implementations to assess its feasibility and scalability.},
keywords = {fastflow, paraphrase, repara},
pubstate = {published},
tppubtype = {article}
}
We introduce a new parallel pattern derived from a specific application domain and show how it turns out to have application beyond its domain of origin. The pool evolution pattern models the parallel evolution of a population subject to mutations and evolving in such a way that a given fitness function is optimized. The pattern has been demonstrated to be suitable for capturing and modeling the parallel patterns underpinning various evolutionary algorithms, as well as other parallel patterns typical of symbolic computation. In this paper we introduce the pattern, we discuss its implementation on modern multi/many core architectures and finally present experimental results obtained with FastFlow and Erlang implementations to assess its feasibility and scalability.
Fabio Tordini, Ivan Merelli, Pietro Liò, Luciano Milanesi, Marco Aldinucci
NuchaRt: embedding high-level parallel computing in R for augmented Hi-C data analysis Book Section
In: Publishing, Springer International (Ed.): Computational Intelligence Methods for Bioinformatics and Biostatistics, vol. 9874, pp. 259–272, Springer International Publishing, Cham (ZG), 2016, ISBN: 978-3-319-44331-7.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, repara
@incollection{15:lnbi:nuchaRt,
title = {NuchaRt: embedding high-level parallel computing in R for augmented Hi-C data analysis},
author = {Fabio Tordini and Ivan Merelli and Pietro Liò and Luciano Milanesi and Marco Aldinucci},
editor = {Springer International Publishing},
url = {https://iris.unito.it/retrieve/handle/2318/1608281/253372/rnuchart.pdf},
doi = {10.1007/978-3-319-44332-4},
isbn = {978-3-319-44331-7},
year = {2016},
date = {2016-01-01},
booktitle = {Computational Intelligence Methods for Bioinformatics and Biostatistics},
volume = {9874},
pages = {259–272},
publisher = {Springer International Publishing},
address = {Cham (ZG)},
series = {Lecture Notes in Computer Science},
abstract = {Recent advances in molecular biology and Bioinformatics techniques brought to an explosion of the information about the spatial organisation of the DNA in the nucleus. High-throughput chromosome conformation capture techniques provide a genome-wide capture of chromatin contacts at unprecedented scales, which permit to identify physical interactions between genetic elements located throughout the human genome. These important studies are hampered by the lack of biologists-friendly software. In this work we present NuchaRt, an R package that wraps NuChart-II, an efficient and highly optimized C++ tool for the exploration of Hi-C data. By rising the level of abstraction, NuchaRt proposes a high-performance pipeline that allows users to orchestrate analysis and visualisation of multi-omics data, making optimal use of the computing capabilities offered by modern multi-core architectures, combined with the versatile and well known R environment for statistical analysis and data visualisation.},
keywords = {bioinformatics, fastflow, repara},
pubstate = {published},
tppubtype = {incollection}
}
Recent advances in molecular biology and Bioinformatics techniques brought to an explosion of the information about the spatial organisation of the DNA in the nucleus. High-throughput chromosome conformation capture techniques provide a genome-wide capture of chromatin contacts at unprecedented scales, which permit to identify physical interactions between genetic elements located throughout the human genome. These important studies are hampered by the lack of biologists-friendly software. In this work we present NuchaRt, an R package that wraps NuChart-II, an efficient and highly optimized C++ tool for the exploration of Hi-C data. By rising the level of abstraction, NuchaRt proposes a high-performance pipeline that allows users to orchestrate analysis and visualisation of multi-omics data, making optimal use of the computing capabilities offered by modern multi-core architectures, combined with the versatile and well known R environment for statistical analysis and data visualisation.
Fabio Tordini
A cloud solution for multi-omics data integration Proceedings Article
In: Proceedings of the 16th IEEE International Conference on Scalable Computing and Communication, pp. 559–566, IEEE Computer Society, 2016, (Best paper award).
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, rephrase
@inproceedings{16:scalcom:cloud,
title = {A cloud solution for multi-omics data integration},
author = {Fabio Tordini},
url = {http://calvados.di.unipi.it/storage/paper_files/2016_cloudpipeline_scalcom.pdf},
doi = {10.1109/UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.131},
year = {2016},
date = {2016-01-01},
booktitle = {Proceedings of the 16th IEEE International Conference on Scalable Computing and Communication},
pages = {559–566},
publisher = {IEEE Computer Society},
abstract = {Recent advances in molecular biology and Bioinformatics techniques have brought to an explosion of the information about the spatial organisation of the DNA inside the nucleus. In particular, 3C-based techniques are revealing the genome folding for many different cell types, and permit to create a more effective representation of the disposition of genes in the three-dimensional space. This information can be used to re-interpret heterogeneous genomic data (multi-omic) relying on 3D maps of the chromosome. The storage and computational requirements needed to accomplish such operations on raw sequenced data have to be fulfilled using HPC solutions, and the the Cloud paradigm is a valuable and convenient mean for delivering HPC to Bioinformatics. In this work we describe a data analysis work-flow that allows the integration and the interpretation of multi-omic data on a sort of ``topographical'' nuclear map, capable of representing the effective disposition of genes in a graph-based representation. We propose a cloud-based task farm pattern to orchestrate the services needed to accomplish genomic data analysis, where each service represents a special-purpose tool, playing a part in well known data analysis pipelines.},
note = {Best paper award},
keywords = {bioinformatics, fastflow, rephrase},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent advances in molecular biology and Bioinformatics techniques have brought to an explosion of the information about the spatial organisation of the DNA inside the nucleus. In particular, 3C-based techniques are revealing the genome folding for many different cell types, and permit to create a more effective representation of the disposition of genes in the three-dimensional space. This information can be used to re-interpret heterogeneous genomic data (multi-omic) relying on 3D maps of the chromosome. The storage and computational requirements needed to accomplish such operations on raw sequenced data have to be fulfilled using HPC solutions, and the the Cloud paradigm is a valuable and convenient mean for delivering HPC to Bioinformatics. In this work we describe a data analysis work-flow that allows the integration and the interpretation of multi-omic data on a sort of ``topographical'' nuclear map, capable of representing the effective disposition of genes in a graph-based representation. We propose a cloud-based task farm pattern to orchestrate the services needed to accomplish genomic data analysis, where each service represents a special-purpose tool, playing a part in well known data analysis pipelines.
Manuel F. Dolz, David Rio Astorga, Javier Fernández, J. Daniel Garc'ıa, Félix Garc'ıa-Carballeira, Marco Danelutto, Massimo Torquati
Embedding Semantics of the Single-Producer/Single-Consumer Lock-Free Queue into a Race Detection Tool Proceedings Article
In: Proceedings of the 7th International Workshop on Programming Models and Applications for Multicores and Manycores, pp. 20–29, ACM, Barcelona, Spain, 2016, ISBN: 978-1-4503-4196-7.
Links | BibTeX | Tags: fastflow, repara
@inproceedings{16:PMAM:SPSC,
title = {Embedding Semantics of the Single-Producer/Single-Consumer Lock-Free Queue into a Race Detection Tool},
author = {Manuel F. Dolz and David Rio Astorga and Javier Fernández and J. Daniel Garc'ıa and Félix Garc'ıa-Carballeira and Marco Danelutto and Massimo Torquati},
url = {https://doi.acm.org/10.1145/2883404.2883406},
doi = {10.1145/2883404.2883406},
isbn = {978-1-4503-4196-7},
year = {2016},
date = {2016-01-01},
booktitle = {Proceedings of the 7th International Workshop on Programming Models and Applications for Multicores and Manycores},
pages = {20–29},
publisher = {ACM},
address = {Barcelona, Spain},
series = {PMAM'16},
keywords = {fastflow, repara},
pubstate = {published},
tppubtype = {inproceedings}
}
Andrea Bracciali, Marco Aldinucci, Murray Patterson, Tobias Marschall, Nadia Pisanti, Ivan Merelli, Massimo Torquati
pWhatsHap: efficient haplotyping for future generation sequencing Journal Article
In: BMC Bioinformatics, vol. 17, no. Suppl 11, pp. 342, 2016.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase, rephrase
@article{16:pwhatshap:bmc,
title = {pWhatsHap: efficient haplotyping for future generation sequencing},
author = {Andrea Bracciali and Marco Aldinucci and Murray Patterson and Tobias Marschall and Nadia Pisanti and Ivan Merelli and Massimo Torquati},
url = {http://bmcbioinformatics.biomedcentral.com/track/pdf/10.1186/s12859-016-1170-y?site=bmcbioinformatics.biomedcentral.com},
doi = {10.1186/s12859-016-1170-y},
year = {2016},
date = {2016-01-01},
journal = {BMC Bioinformatics},
volume = {17},
number = {Suppl 11},
pages = {342},
abstract = {Background: Haplotype phasing is an important problem in the analysis of genomics information. Given a set of DNA fragments of an individual, it consists of determining which one of the possible alleles (alternative forms of a gene) each fragment comes from. Haplotype information is relevant to gene regulation, epigenetics, genome-wide association studies, evolutionary and population studies, and the study of mutations. Haplotyping is currently addressed as an optimisation problem aiming at solutions that minimise, for instance, error correction costs, where costs are a measure of the confidence in the accuracy of the information acquired from DNA sequencing. Solutions have typically an exponential computational complexity. WhatsHap is a recent optimal approach which moves computational complexity from DNA fragment length to fragment overlap, i.e., coverage, and is hence of particular interest when considering sequencing technology's current trends that are producing longer fragments. Results: Given the potential relevance of efficient haplotyping in several analysis pipelines, we have designed and engineered pWhatsHap, a parallel, high-performance version of WhatsHap. pWhatsHap is embedded in a toolkit developed in Python and supports genomics datasets in standard file formats. Building on WhatsHap, pWhatsHap exhibits the same complexity exploring a number of possible solutions which is exponential in the coverage of the dataset. The parallel implementation on multi-core architectures allows for a relevant reduction of the execution time for haplotyping, while the provided results enjoy the same high accuracy as that provided by WhatsHap, which increases with coverage. Conclusions: Due to its structure and management of the large datasets, the parallelisation of WhatsHap posed demanding technical challenges, which have been addressed exploiting a high-level parallel programming framework. The result, pWhatsHap, is a freely available toolkit that improves the efficiency of the analysis of genomics information.},
keywords = {fastflow, paraphrase, rephrase},
pubstate = {published},
tppubtype = {article}
}
Background: Haplotype phasing is an important problem in the analysis of genomics information. Given a set of DNA fragments of an individual, it consists of determining which one of the possible alleles (alternative forms of a gene) each fragment comes from. Haplotype information is relevant to gene regulation, epigenetics, genome-wide association studies, evolutionary and population studies, and the study of mutations. Haplotyping is currently addressed as an optimisation problem aiming at solutions that minimise, for instance, error correction costs, where costs are a measure of the confidence in the accuracy of the information acquired from DNA sequencing. Solutions have typically an exponential computational complexity. WhatsHap is a recent optimal approach which moves computational complexity from DNA fragment length to fragment overlap, i.e., coverage, and is hence of particular interest when considering sequencing technology's current trends that are producing longer fragments. Results: Given the potential relevance of efficient haplotyping in several analysis pipelines, we have designed and engineered pWhatsHap, a parallel, high-performance version of WhatsHap. pWhatsHap is embedded in a toolkit developed in Python and supports genomics datasets in standard file formats. Building on WhatsHap, pWhatsHap exhibits the same complexity exploring a number of possible solutions which is exponential in the coverage of the dataset. The parallel implementation on multi-core architectures allows for a relevant reduction of the execution time for haplotyping, while the provided results enjoy the same high accuracy as that provided by WhatsHap, which increases with coverage. Conclusions: Due to its structure and management of the large datasets, the parallelisation of WhatsHap posed demanding technical challenges, which have been addressed exploiting a high-level parallel programming framework. The result, pWhatsHap, is a freely available toolkit that improves the efficiency of the analysis of genomics information.
2015
Paolo Inaudi
Progettazione e sviluppo di un provider libfabric per la rete ad alte prestazioni Ronniee/A3Cube Masters Thesis
Computer Science Department, University of Torino, 2015.
BibTeX | Tags: fastflow
@mastersthesis{tesi:inaudi:15,
title = {Progettazione e sviluppo di un provider libfabric per la rete ad alte prestazioni Ronniee/A3Cube},
author = {Paolo Inaudi},
year = {2015},
date = {2015-10-01},
school = {Computer Science Department, University of Torino},
keywords = {fastflow},
pubstate = {published},
tppubtype = {mastersthesis}
}
Marco Aldinucci, Marco Danelutto, Maurizio Drocco, Peter Kilpatrick, Guilherme Peretti Pezzi, Massimo Torquati
The Loop-of-Stencil-Reduce paradigm Proceedings Article
In: Proc. of Intl. Workshop on Reengineering for Parallelism in Heterogeneous Parallel Platforms (RePara), pp. 172–177, IEEE, Helsinki, Finland, 2015.
Abstract | Links | BibTeX | Tags: fastflow, HPC, repara
@inproceedings{opencl:ff:ispa:15,
title = {The Loop-of-Stencil-Reduce paradigm},
author = {Marco Aldinucci and Marco Danelutto and Maurizio Drocco and Peter Kilpatrick and Guilherme Peretti Pezzi and Massimo Torquati},
url = {https://iris.unito.it/retrieve/handle/2318/1523738/52857/15_RePara_ISPA.pdf},
doi = {10.1109/Trustcom.2015.628},
year = {2015},
date = {2015-08-01},
booktitle = {Proc. of Intl. Workshop on Reengineering for Parallelism in Heterogeneous Parallel Platforms (RePara)},
pages = {172–177},
publisher = {IEEE},
address = {Helsinki, Finland},
abstract = {In this paper we advocate the Loop-of-stencil-reduce pattern as a way to simplify the parallel programming of heterogeneous platforms (multicore+GPUs). Loop-of-Stencil-reduce is general enough to subsume map, reduce, map-reduce, stencil, stencil-reduce, and, crucially, their usage in a loop. It transparently targets (by using OpenCL) combinations of CPU cores and GPUs, and it makes it possible to simplify the deployment of a single stencil computation kernel on different GPUs. The paper discusses the implementation of Loop-of-stencil-reduce within the FastFlow parallel framework, considering a simple iterative data-parallel application as running example (Game of Life) and a highly effective parallel filter for visual data restoration to assess performance. Thanks to the high-level design of the Loop-of-stencil-reduce, it was possible to run the filter seamlessly on a multicore machine, on multi-GPUs, and on both.},
keywords = {fastflow, HPC, repara},
pubstate = {published},
tppubtype = {inproceedings}
}
In this paper we advocate the Loop-of-stencil-reduce pattern as a way to simplify the parallel programming of heterogeneous platforms (multicore+GPUs). Loop-of-Stencil-reduce is general enough to subsume map, reduce, map-reduce, stencil, stencil-reduce, and, crucially, their usage in a loop. It transparently targets (by using OpenCL) combinations of CPU cores and GPUs, and it makes it possible to simplify the deployment of a single stencil computation kernel on different GPUs. The paper discusses the implementation of Loop-of-stencil-reduce within the FastFlow parallel framework, considering a simple iterative data-parallel application as running example (Game of Life) and a highly effective parallel filter for visual data restoration to assess performance. Thanks to the high-level design of the Loop-of-stencil-reduce, it was possible to run the filter seamlessly on a multicore machine, on multi-GPUs, and on both.
Fabio Tordini, Maurizio Drocco, Ivan Merelli, Luciano Milanesi, Pietro Liò, Marco Aldinucci
NuChart-II: a graph-based approach for the analysis and interpretation of Hi-C data Proceedings Article
In: Serio, Clelia Di, Liò, Pietro, Nonis, Alessandro, Tagliaferri, Roberto (Ed.): Proc. of 11th Intl. Meeting on Computational Intelligence Methods for Bioinformatics and Biostatistics (CIBB), pp. 298–311, Springer, Cambridge, UK, 2015, ISBN: 978-3-319-24461-7.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, paraphrase, repara
@inproceedings{14:ff:nuchart:cibb,
title = {NuChart-II: a graph-based approach for the analysis and interpretation of Hi-C data},
author = {Fabio Tordini and Maurizio Drocco and Ivan Merelli and Luciano Milanesi and Pietro Liò and Marco Aldinucci},
editor = {Clelia Di Serio and Pietro Liò and Alessandro Nonis and Roberto Tagliaferri},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_nuchart_cibb.pdf},
doi = {10.1007/978-3-319-24462-4_25},
isbn = {978-3-319-24461-7},
year = {2015},
date = {2015-06-01},
booktitle = {Proc. of 11th Intl. Meeting on Computational Intelligence Methods for Bioinformatics and Biostatistics (CIBB)},
volume = {8623},
pages = {298–311},
publisher = {Springer},
address = {Cambridge, UK},
series = {LNCS},
abstract = {Long-range chromosomal associations between genomic regions, and their repositioning in the 3D space of the nucleus, are now considered to be key contributors to the regulation of gene expressions, and important links have been highlighted with other genomic features involved in DNA rearrangements. Recent Chromosome Conformation Capture (3C) measurements performed with high throughput sequencing (Hi-C) and molecular dynamics studies show that there is a large correlation between co-localization and co-regulation of genes, but these important researches are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we present NuChart-II, a software that allows the user to annotate and visualize a list of input genes with information relying on Hi-C data, integrating knowledge data about genomic features that are involved in the chromosome spatial organization. This software works directly with sequenced reads to identify related Hi-C fragments, with the aim of creating gene-centric neighbourhood graphs on which multi-omics features can be mapped. NuChart-II is a highly optimized implementation of a previous prototype package developed in R, in which the graph-based representation of Hi-C data was tested. The prototype showed inevitable problems of scalability while working genome-wide on large datasets: particular attention has been paid in optimizing the data structures employed while constructing the neighbourhood graph, so as to foster an efficient parallel implementation of the software. The normalization of Hi-C data has been modified and improved, in order to provide a reliable estimation of proximity likelihood for the genes.},
keywords = {bioinformatics, fastflow, paraphrase, repara},
pubstate = {published},
tppubtype = {inproceedings}
}
Long-range chromosomal associations between genomic regions, and their repositioning in the 3D space of the nucleus, are now considered to be key contributors to the regulation of gene expressions, and important links have been highlighted with other genomic features involved in DNA rearrangements. Recent Chromosome Conformation Capture (3C) measurements performed with high throughput sequencing (Hi-C) and molecular dynamics studies show that there is a large correlation between co-localization and co-regulation of genes, but these important researches are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we present NuChart-II, a software that allows the user to annotate and visualize a list of input genes with information relying on Hi-C data, integrating knowledge data about genomic features that are involved in the chromosome spatial organization. This software works directly with sequenced reads to identify related Hi-C fragments, with the aim of creating gene-centric neighbourhood graphs on which multi-omics features can be mapped. NuChart-II is a highly optimized implementation of a previous prototype package developed in R, in which the graph-based representation of Hi-C data was tested. The prototype showed inevitable problems of scalability while working genome-wide on large datasets: particular attention has been paid in optimizing the data structures employed while constructing the neighbourhood graph, so as to foster an efficient parallel implementation of the software. The normalization of Hi-C data has been modified and improved, in order to provide a reliable estimation of proximity likelihood for the genes.
Maurizio Drocco, Claudia Misale, Guilherme Peretti Pezzi, Fabio Tordini, Marco Aldinucci
Memory-Optimised Parallel Processing of Hi-C Data Proceedings Article
In: Proc. of 23rd Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP), pp. 1–8, IEEE, 2015.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, impact, paraphrase, repara
@inproceedings{nuchart:speedup:15,
title = {Memory-Optimised Parallel Processing of Hi-C Data},
author = {Maurizio Drocco and Claudia Misale and Guilherme Peretti Pezzi and Fabio Tordini and Marco Aldinucci},
url = {https://iris.unito.it/retrieve/handle/2318/1521910/40615/2015_pdp_memopt.pdf},
doi = {10.1109/PDP.2015.63},
year = {2015},
date = {2015-03-01},
booktitle = {Proc. of 23rd Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)},
pages = {1–8},
publisher = {IEEE},
abstract = {This paper presents the optimisation efforts on the creation of a graph-based mapping representation of gene adjacency. The method is based on the Hi-C process, starting from Next Generation Sequencing data, and it analyses a huge amount of static data in order to produce maps for one or more genes. Straightforward parallelisation of this scheme does not yield acceptable performance on multicore architectures since the scalability is rather limited due to the memory bound nature of the problem. This work focuses on the memory optimisations that can be applied to the graph construction algorithm and its (complex) data structures to derive a cache-oblivious algorithm and eventually to improve the memory bandwidth utilisation. We used as running example NuChart-II, a tool for annotation and statistic analysis of Hi-C data that creates a gene-centric neighborhood graph. The proposed approach, which is exemplified for Hi-C, addresses several common issue in the parallelisation of memory bound algorithms for multicore. Results show that the proposed approach is able to increase the parallel speedup from 7x to 22x (on a 32-core platform). Finally, the proposed C++ implementation outperforms the first R NuChart prototype, by which it was not possible to complete the graph generation because of strong memory-saturation problems.},
keywords = {bioinformatics, fastflow, impact, paraphrase, repara},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper presents the optimisation efforts on the creation of a graph-based mapping representation of gene adjacency. The method is based on the Hi-C process, starting from Next Generation Sequencing data, and it analyses a huge amount of static data in order to produce maps for one or more genes. Straightforward parallelisation of this scheme does not yield acceptable performance on multicore architectures since the scalability is rather limited due to the memory bound nature of the problem. This work focuses on the memory optimisations that can be applied to the graph construction algorithm and its (complex) data structures to derive a cache-oblivious algorithm and eventually to improve the memory bandwidth utilisation. We used as running example NuChart-II, a tool for annotation and statistic analysis of Hi-C data that creates a gene-centric neighborhood graph. The proposed approach, which is exemplified for Hi-C, addresses several common issue in the parallelisation of memory bound algorithms for multicore. Results show that the proposed approach is able to increase the parallel speedup from 7x to 22x (on a 32-core platform). Finally, the proposed C++ implementation outperforms the first R NuChart prototype, by which it was not possible to complete the graph generation because of strong memory-saturation problems.
Fabio Tordini, Maurizio Drocco, Claudia Misale, Luciano Milanesi, Pietro Liò, Ivan Merelli, Marco Aldinucci
Parallel Exploration of the Nuclear Chromosome Conformation with NuChart-II Proceedings Article
In: Proc. of 23rd Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP), IEEE, 2015.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, impact, paraphrase, repara
@inproceedings{nuchar:tool:15,
title = {Parallel Exploration of the Nuclear Chromosome Conformation with NuChart-II},
author = {Fabio Tordini and Maurizio Drocco and Claudia Misale and Luciano Milanesi and Pietro Liò and Ivan Merelli and Marco Aldinucci},
url = {https://iris.unito.it/retrieve/handle/2318/1522038/40619/2015_pdp_nuchartff.pdf},
doi = {10.1109/PDP.2015.104},
year = {2015},
date = {2015-03-01},
booktitle = {Proc. of 23rd Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)},
publisher = {IEEE},
abstract = {High-throughput molecular biology techniques are widely used to identify physical interactions between genetic elements located throughout the human genome. Chromosome Conformation Capture (3C) and other related techniques allow to investigate the spatial organisation of chromosomes in the cell's natural state. Recent results have shown that there is a large correlation between co-localization and co-regulation of genes, but these important information are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we introduce NuChart-II, a tool for Hi-C data analysis that provides a gene-centric view of the chromosomal neighbour- hood in a graph-based manner. NuChart-II is an efficient and highly optimized C++ re-implementation of a previous prototype package developed in R. Representing Hi-C data using a graph-based approach overcomes the common view relying on genomic coordinates and permits the use of graph analysis techniques to explore the spatial conformation of a gene neighbourhood.},
keywords = {bioinformatics, fastflow, impact, paraphrase, repara},
pubstate = {published},
tppubtype = {inproceedings}
}
High-throughput molecular biology techniques are widely used to identify physical interactions between genetic elements located throughout the human genome. Chromosome Conformation Capture (3C) and other related techniques allow to investigate the spatial organisation of chromosomes in the cell's natural state. Recent results have shown that there is a large correlation between co-localization and co-regulation of genes, but these important information are hampered by the lack of biologists-friendly analysis and visualisation software. In this work we introduce NuChart-II, a tool for Hi-C data analysis that provides a gene-centric view of the chromosomal neighbour- hood in a graph-based manner. NuChart-II is an efficient and highly optimized C++ re-implementation of a previous prototype package developed in R. Representing Hi-C data using a graph-based approach overcomes the common view relying on genomic coordinates and permits the use of graph analysis techniques to explore the spatial conformation of a gene neighbourhood.
Paolo Viviani
Parallel Computing Techniques for High Energy Physics Masters Thesis
Physics Department, University of Torino, 2015.
Abstract | BibTeX | Tags: fastflow, impact
@mastersthesis{tesi:viviani:15,
title = {Parallel Computing Techniques for High Energy Physics},
author = {Paolo Viviani},
year = {2015},
date = {2015-01-01},
school = {Physics Department, University of Torino},
abstract = {Modern experimental achievements, with LHC results as a prominent but not exclusive representative, have undisclosed a new range of challenges concerning theoretical com- putations. Tree level QED calculation are no more satisfactory due to the very small experimental uncertainty of precision e+ e- measurements, so Next To Leading and Next to Next to Leading Order calculations are required. At the same time many-legs, high-order QCD processes needed to simulate LHC events are raising even more the bar of computational complexity. The drive for the present work has been the interest in calculating high multiplicity Higgs boson processes with a dedicated software library (RECOLA) currently under development at the University of Torino, as well as the related technological challenges. This thesis undertakes the task of exploring the possibilities offered by present and upcoming computing technologies in order to face these challenges properly. The first two chapters outlines the theoretical context and the available technologies. In chapter 3 a a case study is examined in full detail, in order to explore the suitability of different parallel computing solutions. In the chapter 4, some of those solutions are implemented in the context of the RECOLA library, allowing it to handle processes at a previously unexplored scale of complexity. Alongside, the potential of new, cost-effective parallel architectures is tested.},
keywords = {fastflow, impact},
pubstate = {published},
tppubtype = {mastersthesis}
}
Modern experimental achievements, with LHC results as a prominent but not exclusive representative, have undisclosed a new range of challenges concerning theoretical com- putations. Tree level QED calculation are no more satisfactory due to the very small experimental uncertainty of precision e+ e- measurements, so Next To Leading and Next to Next to Leading Order calculations are required. At the same time many-legs, high-order QCD processes needed to simulate LHC events are raising even more the bar of computational complexity. The drive for the present work has been the interest in calculating high multiplicity Higgs boson processes with a dedicated software library (RECOLA) currently under development at the University of Torino, as well as the related technological challenges. This thesis undertakes the task of exploring the possibilities offered by present and upcoming computing technologies in order to face these challenges properly. The first two chapters outlines the theoretical context and the available technologies. In chapter 3 a a case study is examined in full detail, in order to explore the suitability of different parallel computing solutions. In the chapter 4, some of those solutions are implemented in the context of the RECOLA library, allowing it to handle processes at a previously unexplored scale of complexity. Alongside, the potential of new, cost-effective parallel architectures is tested.
Ivan Merelli, Fabio Tordini, Maurizio Drocco, Marco Aldinucci, Pietro Liò, Luciano Milanesi
Integrating Multi-omic features exploiting Chromosome Conformation Capture data Journal Article
In: Frontiers in Genetics, vol. 6, no. 40, 2015, ISSN: 1664-8021.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow
@article{nuchart:frontiers:15,
title = {Integrating Multi-omic features exploiting Chromosome Conformation Capture data},
author = {Ivan Merelli and Fabio Tordini and Maurizio Drocco and Marco Aldinucci and Pietro Liò and Luciano Milanesi},
url = {http://journal.frontiersin.org/Journal/10.3389/fgene.2015.00040/pdf},
doi = {10.3389/fgene.2015.00040},
issn = {1664-8021},
year = {2015},
date = {2015-01-01},
journal = {Frontiers in Genetics},
volume = {6},
number = {40},
abstract = {The representation, integration and interpretation of omic data is a complex task, in particular considering the huge amount of information that is daily produced in molecular biology laboratories all around the world. The reason is that sequencing data regarding expression profiles, methylation patterns, and chromatin domains is difficult to harmonize in a systems biology view, since genome browsers only allow coordinate-based representations, discarding functional clusters created by the spatial conformation of the DNA in the nucleus. In this context, recent progresses in high throughput molecular biology techniques and bioinformatics have provided insights into chromatin interactions on a larger scale and offer a formidable support for the interpretation of multi-omic data. In particular, a novel sequencing technique called Chromosome Conformation Capture (3C) allows the analysis of the chromosome organization in the cell's natural state. While performed genome wide, this technique is usually called Hi-C. Inspired by service applications such as Google Maps, we developed NuChart, an R package that integrates Hi-C data to describe the chromosomal neighbourhood starting from the information about gene positions, with the possibility of mapping on the achieved graphs genomic features such as methylation patterns and histone modifications, along with expression profiles. In this paper we show the importance of the NuChart application for the integration of multi-omic data in a systems biology fashion, with particular interest in cytogenetic applications of these techniques. Moreover, we demonstrate how the integration of multi-omic data can provide useful information in understanding why genes are in certain specific positions inside the nucleus and how epigenetic patterns correlate with their expression.},
keywords = {bioinformatics, fastflow},
pubstate = {published},
tppubtype = {article}
}
The representation, integration and interpretation of omic data is a complex task, in particular considering the huge amount of information that is daily produced in molecular biology laboratories all around the world. The reason is that sequencing data regarding expression profiles, methylation patterns, and chromatin domains is difficult to harmonize in a systems biology view, since genome browsers only allow coordinate-based representations, discarding functional clusters created by the spatial conformation of the DNA in the nucleus. In this context, recent progresses in high throughput molecular biology techniques and bioinformatics have provided insights into chromatin interactions on a larger scale and offer a formidable support for the interpretation of multi-omic data. In particular, a novel sequencing technique called Chromosome Conformation Capture (3C) allows the analysis of the chromosome organization in the cell's natural state. While performed genome wide, this technique is usually called Hi-C. Inspired by service applications such as Google Maps, we developed NuChart, an R package that integrates Hi-C data to describe the chromosomal neighbourhood starting from the information about gene positions, with the possibility of mapping on the achieved graphs genomic features such as methylation patterns and histone modifications, along with expression profiles. In this paper we show the importance of the NuChart application for the integration of multi-omic data in a systems biology fashion, with particular interest in cytogenetic applications of these techniques. Moreover, we demonstrate how the integration of multi-omic data can provide useful information in understanding why genes are in certain specific positions inside the nucleus and how epigenetic patterns correlate with their expression.
Marco Aldinucci, Andrea Bracciali, Tobias Marschall, Murray Patterson, Nadia Pisanti, Massimo Torquati
High-Performance Haplotype Assembly Proceedings Article
In: Serio, Clelia Di, Liò, Pietro, Nonis, Alessandro, Tagliaferri, Roberto (Ed.): Computational Intelligence Methods for Bioinformatics and Biostatistics - 11th International Meeting, CIBB 2014, Cambridge, UK, June 26-28, 2014, Revised Selected Papers, pp. 245–258, Springer, Cambridge, UK, 2015.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow
@inproceedings{14:ff:whatsapp:cibb,
title = {High-Performance Haplotype Assembly},
author = {Marco Aldinucci and Andrea Bracciali and Tobias Marschall and Murray Patterson and Nadia Pisanti and Massimo Torquati},
editor = {Clelia Di Serio and Pietro Liò and Alessandro Nonis and Roberto Tagliaferri},
url = {https://iris.unito.it/retrieve/handle/2318/1523292/46714/2014_pHaplo_cibb.pdf},
doi = {10.1007/978-3-319-24462-4_21},
year = {2015},
date = {2015-01-01},
booktitle = {Computational Intelligence Methods for Bioinformatics and Biostatistics - 11th International Meeting, CIBB 2014, Cambridge, UK, June 26-28, 2014, Revised Selected Papers},
volume = {8623},
pages = {245–258},
publisher = {Springer},
address = {Cambridge, UK},
series = {LNCS},
abstract = {The problem of Haplotype Assembly is an essential step in human genome analysis. It is typically formalised as the Minimum Error Correction (MEC) problem which is NP-hard. MEC has been approached using heuristics, integer linear programming, and fixed-parameter tractability (FPT), including approaches whose runtime is exponential in the length of the DNA fragments obtained by the sequencing process. Technological improvements are currently increasing fragment length, which drastically elevates computational costs for such methods. We present pWhatsHap, a multi-core parallelisation of WhatsHap, a recent FPT optimal approach to MEC. WhatsHap moves complexity from fragment length to fragment overlap and is hence of particular interest when considering sequencing technology's current trends. pWhatsHap further improves the efficiency in solving the MEC problem, as shown by experiments performed on datasets with high coverage.},
keywords = {bioinformatics, fastflow},
pubstate = {published},
tppubtype = {inproceedings}
}
The problem of Haplotype Assembly is an essential step in human genome analysis. It is typically formalised as the Minimum Error Correction (MEC) problem which is NP-hard. MEC has been approached using heuristics, integer linear programming, and fixed-parameter tractability (FPT), including approaches whose runtime is exponential in the length of the DNA fragments obtained by the sequencing process. Technological improvements are currently increasing fragment length, which drastically elevates computational costs for such methods. We present pWhatsHap, a multi-core parallelisation of WhatsHap, a recent FPT optimal approach to MEC. WhatsHap moves complexity from fragment length to fragment overlap and is hence of particular interest when considering sequencing technology's current trends. pWhatsHap further improves the efficiency in solving the MEC problem, as shown by experiments performed on datasets with high coverage.
Marco Aldinucci, Guilherme Peretti Pezzi, Maurizio Drocco, Concetto Spampinato, Massimo Torquati
Parallel Visual Data Restoration on Multi-GPGPUs using Stencil-Reduce Pattern Journal Article
In: International Journal of High Performance Computing Applications, vol. 29, no. 4, pp. 461–472, 2015.
Abstract | Links | BibTeX | Tags: fastflow, HPC, impact, paraphrase
@article{ff:denoiser:ijhpca:15,
title = {Parallel Visual Data Restoration on Multi-GPGPUs using Stencil-Reduce Pattern},
author = {Marco Aldinucci and Guilherme Peretti Pezzi and Maurizio Drocco and Concetto Spampinato and Massimo Torquati},
url = {https://iris.unito.it/retrieve/handle/2318/1522073/299200/ijhpca_4aperto.pdf},
doi = {10.1177/1094342014567907},
year = {2015},
date = {2015-01-01},
journal = {International Journal of High Performance Computing Applications},
volume = {29},
number = {4},
pages = {461–472},
abstract = {In this paper, a highly effective parallel filter for visual data restoration is presented. The filter is designed following a skeletal approach, using a newly proposed stencil-reduce, and has been implemented by way of the FastFlow parallel programming library. As a result of its high-level design, it is possible to run the filter seamlessly on a multicore machine, on multi-GPGPUs, or on both. The design and implementation of the filter are discussed, and an experimental evaluation is presented.},
keywords = {fastflow, HPC, impact, paraphrase},
pubstate = {published},
tppubtype = {article}
}
In this paper, a highly effective parallel filter for visual data restoration is presented. The filter is designed following a skeletal approach, using a newly proposed stencil-reduce, and has been implemented by way of the FastFlow parallel programming library. As a result of its high-level design, it is possible to run the filter seamlessly on a multicore machine, on multi-GPGPUs, or on both. The design and implementation of the filter are discussed, and an experimental evaluation is presented.
2014
Marco Aldinucci, Sonia Campa, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
Pool evolution: a domain specific parallel pattern Proceedings Article
In: Proc.of the 7th Intl. Symposium on High-level Parallel Programming and Applications (HLPP), Amsterdam, The Netherlands, 2014.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase, repara
@inproceedings{2014:ff:pool:hlpp,
title = {Pool evolution: a domain specific parallel pattern},
author = {Marco Aldinucci and Sonia Campa and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_hlpp_pool.pdf},
year = {2014},
date = {2014-07-01},
booktitle = {Proc.of the 7th Intl. Symposium on High-level Parallel Programming and Applications (HLPP)},
address = {Amsterdam, The Netherlands},
abstract = {We introduce a new parallel pattern derived from a specific application domain and show how it turns out to have application beyond its domain of origin. The pool evolution pattern models the parallel evolution of a population subject to mutations and evolving in such a way that a given fitness function is optimized. The pattern has been demonstrated to be suitable for capturing and modeling the parallel patterns underpinning various evolutionary algorithms, as well as other parallel patterns typical of symbolic computation. In this paper we introduce the pattern, developed in the framework of the ParaPhrase EU-funded FP7 project, we discuss its implementation on modern multi/many core architectures and finally present experimental results obtained with FastFlow and Erlang implementations to assess its feasibility and scalability.},
keywords = {fastflow, paraphrase, repara},
pubstate = {published},
tppubtype = {inproceedings}
}
We introduce a new parallel pattern derived from a specific application domain and show how it turns out to have application beyond its domain of origin. The pool evolution pattern models the parallel evolution of a population subject to mutations and evolving in such a way that a given fitness function is optimized. The pattern has been demonstrated to be suitable for capturing and modeling the parallel patterns underpinning various evolutionary algorithms, as well as other parallel patterns typical of symbolic computation. In this paper we introduce the pattern, developed in the framework of the ParaPhrase EU-funded FP7 project, we discuss its implementation on modern multi/many core architectures and finally present experimental results obtained with FastFlow and Erlang implementations to assess its feasibility and scalability.
Marco Aldinucci, Massimo Torquati, Maurizio Drocco, Guilherme Peretti Pezzi, Concetto Spampinato
FastFlow: Combining Pattern-Level Abstraction and Efficiency in GPGPUs Proceedings Article
In: GPU Technology Conference (GTC), San Jose, CA, USA, 2014.
Abstract | Links | BibTeX | Tags: fastflow, HPC, impact, paraphrase
@inproceedings{ff:gtc:2014,
title = {FastFlow: Combining Pattern-Level Abstraction and Efficiency in GPGPUs},
author = {Marco Aldinucci and Massimo Torquati and Maurizio Drocco and Guilherme Peretti Pezzi and Concetto Spampinato},
url = {http://calvados.di.unipi.it/storage/talks/2014_S4729-Marco-Aldinucci.pdf},
year = {2014},
date = {2014-03-01},
booktitle = {GPU Technology Conference (GTC)},
address = {San Jose, CA, USA},
abstract = {Learn how FastFlow's parallel patterns can be used to design parallel applications for execution on both CPUs and GPGPUs while avoiding most of the complex low-level detail needed to make them efficient, portable and rapid to prototype. As use case, we will show the design and effectiveness of a novel universal image filtering template based on the variational approach.},
keywords = {fastflow, HPC, impact, paraphrase},
pubstate = {published},
tppubtype = {inproceedings}
}
Learn how FastFlow's parallel patterns can be used to design parallel applications for execution on both CPUs and GPGPUs while avoiding most of the complex low-level detail needed to make them efficient, portable and rapid to prototype. As use case, we will show the design and effectiveness of a novel universal image filtering template based on the variational approach.
Marco Aldinucci, Massimo Torquati, Maurizio Drocco, Guilherme Peretti Pezzi, Concetto Spampinato
An Overview of FastFlow: Combining Pattern-Level Abstraction and Efficiency in GPGPUs Proceedings Article
In: GPU Technology Conference (GTC), San Jose, CA, USA, 2014.
Abstract | Links | BibTeX | Tags: fastflow, HPC, impact, paraphrase
@inproceedings{ff:gtc:2014:short,
title = {An Overview of FastFlow: Combining Pattern-Level Abstraction and Efficiency in GPGPUs},
author = {Marco Aldinucci and Massimo Torquati and Maurizio Drocco and Guilherme Peretti Pezzi and Concetto Spampinato},
url = {http://calvados.di.unipi.it/storage/talks/2014_S4585-Marco-Aldinucci.pdf},
year = {2014},
date = {2014-03-01},
booktitle = {GPU Technology Conference (GTC)},
address = {San Jose, CA, USA},
abstract = {Get an overview of FastFlow's parallel patterns can be used to design parallel applications for execution on both CPUs and GPGPUs while avoiding most of the complex low-level detail needed to make them efficient, portable and rapid to prototype. For a more detailed and technical review of FastFlow's parallel patterns as well as a use case where we will show the design and effectiveness of a novel universal image filtering template based on the variational approach.},
keywords = {fastflow, HPC, impact, paraphrase},
pubstate = {published},
tppubtype = {inproceedings}
}
Get an overview of FastFlow's parallel patterns can be used to design parallel applications for execution on both CPUs and GPGPUs while avoiding most of the complex low-level detail needed to make them efficient, portable and rapid to prototype. For a more detailed and technical review of FastFlow's parallel patterns as well as a use case where we will show the design and effectiveness of a novel universal image filtering template based on the variational approach.
Daniele Buono, Marco Danelutto, Tiziano De Matteis, Gabriele Mencagli, Massimo Torquati
A Lightweight Run-Time Support For Fast Dense Linear Algebra on Multi-Core Proceedings Article
In: Proc. of the 12th International Conference on Parallel and Distributed Computing and Networks (PDCN 2014), IASTED, ACTA press, 2014.
BibTeX | Tags: fastflow
@inproceedings{ff:ffmdf:pdcn:14,
title = {A Lightweight Run-Time Support For Fast Dense Linear Algebra on Multi-Core},
author = {Daniele Buono and Marco Danelutto and Tiziano De Matteis and Gabriele Mencagli and Massimo Torquati},
year = {2014},
date = {2014-02-01},
booktitle = {Proc. of the 12th International Conference on Parallel and Distributed Computing and Networks (PDCN 2014)},
publisher = {IASTED, ACTA press},
keywords = {fastflow},
pubstate = {published},
tppubtype = {inproceedings}
}
Marco Aldinucci, Massimo Torquati, Concetto Spampinato, Maurizio Drocco, Claudia Misale, Cristina Calcagno, Mario Coppo
Parallel stochastic systems biology in the cloud Journal Article
In: Briefings in Bioinformatics, vol. 15, no. 5, pp. 798–813, 2014, ISSN: 1467-5463.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, impact, paraphrase
@article{cwc:cloud:bib:13,
title = {Parallel stochastic systems biology in the cloud},
author = {Marco Aldinucci and Massimo Torquati and Concetto Spampinato and Maurizio Drocco and Claudia Misale and Cristina Calcagno and Mario Coppo},
url = {https://iris.unito.it/retrieve/handle/2318/140080/22528/FF_Cloud_briefings_final_submitted_copy.pdf},
doi = {10.1093/bib/bbt040},
issn = {1467-5463},
year = {2014},
date = {2014-01-01},
journal = {Briefings in Bioinformatics},
volume = {15},
number = {5},
pages = {798–813},
abstract = {The stochastic modelling of biological systems, coupled with Monte Carlo simulation of models, is an increasingly popular technique in bioinformatics. The simulation-analysis workflow may result computationally expensive reducing the interactivity required in the model tuning. In this work, we advocate the high-level software design as a vehicle for building efficient and portable parallel simulators for the cloud. In particular, the Calculus of Wrapped Components (CWC) simulator for systems biology, which is designed according to the FastFlow pattern-based approach, is presented and discussed. Thanks to the FastFlow framework, the CWC simulator is designed as a high-level workflow that can simulate CWC models, merge simulation results and statistically analyse them in a single parallel workflow in the cloud. To improve interactivity, successive phases are pipelined in such a way that the workflow begins to output a stream of analysis results immediately after simulation is started. Performance and effectiveness of the CWC simulator are validated on the Amazon Elastic Compute Cloud.},
keywords = {bioinformatics, fastflow, impact, paraphrase},
pubstate = {published},
tppubtype = {article}
}
The stochastic modelling of biological systems, coupled with Monte Carlo simulation of models, is an increasingly popular technique in bioinformatics. The simulation-analysis workflow may result computationally expensive reducing the interactivity required in the model tuning. In this work, we advocate the high-level software design as a vehicle for building efficient and portable parallel simulators for the cloud. In particular, the Calculus of Wrapped Components (CWC) simulator for systems biology, which is designed according to the FastFlow pattern-based approach, is presented and discussed. Thanks to the FastFlow framework, the CWC simulator is designed as a high-level workflow that can simulate CWC models, merge simulation results and statistically analyse them in a single parallel workflow in the cloud. To improve interactivity, successive phases are pipelined in such a way that the workflow begins to output a stream of analysis results immediately after simulation is started. Performance and effectiveness of the CWC simulator are validated on the Amazon Elastic Compute Cloud.
Marco Aldinucci, Sonia Campa, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
Design patterns percolating to parallel programming framework implementation Journal Article
In: International Journal of Parallel Programming, vol. 42, no. 6, pp. 1012–1031, 2014, ISSN: 0885-7458.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase
@article{ijpp:patterns:13,
title = {Design patterns percolating to parallel programming framework implementation},
author = {Marco Aldinucci and Sonia Campa and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
url = {https://iris.unito.it/retrieve/handle/2318/140069/22527/2013_ijpp_patterns-web_4aperto_1238811.pdf},
doi = {10.1007/s10766-013-0273-6},
issn = {0885-7458},
year = {2014},
date = {2014-01-01},
journal = {International Journal of Parallel Programming},
volume = {42},
number = {6},
pages = {1012–1031},
abstract = {Structured parallel programming is recognised as a viable and effective means of tackling parallel programming problems. Recently, a set of simple and powerful parallel building blocks (RISC-pb2l) has been proposed to support modelling and implementation of parallel frameworks. In this work we demonstrate how that same parallel building block set may be used to model both general purpose parallel programming abstractions, not usually listed in classical skeleton sets, and more specialized domain specific parallel patterns. We show how an implementation of RISC-pb2l can be realised via the FastFlow framework and present experimental evidence of the feasibility and efficiency of the approach.},
keywords = {fastflow, paraphrase},
pubstate = {published},
tppubtype = {article}
}
Structured parallel programming is recognised as a viable and effective means of tackling parallel programming problems. Recently, a set of simple and powerful parallel building blocks (RISC-pb2l) has been proposed to support modelling and implementation of parallel frameworks. In this work we demonstrate how that same parallel building block set may be used to model both general purpose parallel programming abstractions, not usually listed in classical skeleton sets, and more specialized domain specific parallel patterns. We show how an implementation of RISC-pb2l can be realised via the FastFlow framework and present experimental evidence of the feasibility and efficiency of the approach.
Marco Aldinucci, Salvatore Ruggieri, Massimo Torquati
Decision Tree Building on Multi-Core using FastFlow Journal Article
In: Concurrency and Computation: Practice and Experience, vol. 26, no. 3, pp. 800–820, 2014.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase
@article{yadtff:ccpe:13,
title = {Decision Tree Building on Multi-Core using FastFlow},
author = {Marco Aldinucci and Salvatore Ruggieri and Massimo Torquati},
url = {https://iris.unito.it/retrieve/handle/2318/139522/118602/yadtff-j.pdf},
doi = {10.1002/cpe.3063},
year = {2014},
date = {2014-01-01},
journal = {Concurrency and Computation: Practice and Experience},
volume = {26},
number = {3},
pages = {800–820},
abstract = {The whole computer hardware industry embraced multi-core. The extreme optimisation of sequential algorithms is then no longer sufficient to squeeze the real machine power, which can be only exploited via thread-level parallelism. Decision tree algorithms exhibit natural concurrency that makes them suitable to be parallelised. This paper presents an in-depth study of the parallelisation of an implementation of the C4.5 algorithm for multi-core architectures. We characterise elapsed time lower bounds for the forms of parallelisations adopted, and achieve close to optimal performances. Our implementation is based on the FastFlow parallel programming environment and it requires minimal changes to the original sequential code.},
keywords = {fastflow, paraphrase},
pubstate = {published},
tppubtype = {article}
}
The whole computer hardware industry embraced multi-core. The extreme optimisation of sequential algorithms is then no longer sufficient to squeeze the real machine power, which can be only exploited via thread-level parallelism. Decision tree algorithms exhibit natural concurrency that makes them suitable to be parallelised. This paper presents an in-depth study of the parallelisation of an implementation of the C4.5 algorithm for multi-core architectures. We characterise elapsed time lower bounds for the forms of parallelisations adopted, and achieve close to optimal performances. Our implementation is based on the FastFlow parallel programming environment and it requires minimal changes to the original sequential code.
Marco Aldinucci, Cristina Calcagno, Mario Coppo, Ferruccio Damiani, Maurizio Drocco, Eva Sciacca, Salvatore Spinella, Massimo Torquati, Angelo Troina
On designing multicore-aware simulators for systems biology endowed with on-line statistics Journal Article
In: BioMed Research International, 2014.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, paraphrase
@article{cwcsim:ff:multicore:biomed:14,
title = {On designing multicore-aware simulators for systems biology endowed with on-line statistics},
author = {Marco Aldinucci and Cristina Calcagno and Mario Coppo and Ferruccio Damiani and Maurizio Drocco and Eva Sciacca and Salvatore Spinella and Massimo Torquati and Angelo Troina},
url = {http://downloads.hindawi.com/journals/bmri/2014/207041.pdf},
doi = {10.1155/2014/207041},
year = {2014},
date = {2014-01-01},
journal = {BioMed Research International},
abstract = {The paper arguments are on enabling methodologies for the design of a fully parallel, online, interactive tool aiming to support the bioinformatics scientists .In particular, the features of these methodologies, supported by the FastFlow parallel programming framework, are shown on a simulation tool to perform the modeling, the tuning, and the sensitivity analysis of stochastic biological models. A stochastic simulation needs thousands of independent simulation trajectories turning into big data that should be analysed by statistic and data mining tools. In the considered approach the two stages are pipelined in such a way that the simulation stage streams out the partial results of all simulation trajectories to the analysis stage that immediately produces a partial result. The simulation-analysis workflow is validated for performance and effectiveness of the online analysis in capturing biological systems behavior on a multicore platform and representative proof-of-concept biological systems. The exploited methodologies include pattern-based parallel programming and data streaming that provide key features to the software designers such as performance portability and efficient in-memory (big) data management and movement. Two paradigmatic classes of biological systems exhibiting multistable and oscillatory behavior are used as a testbed.},
keywords = {bioinformatics, fastflow, paraphrase},
pubstate = {published},
tppubtype = {article}
}
The paper arguments are on enabling methodologies for the design of a fully parallel, online, interactive tool aiming to support the bioinformatics scientists .In particular, the features of these methodologies, supported by the FastFlow parallel programming framework, are shown on a simulation tool to perform the modeling, the tuning, and the sensitivity analysis of stochastic biological models. A stochastic simulation needs thousands of independent simulation trajectories turning into big data that should be analysed by statistic and data mining tools. In the considered approach the two stages are pipelined in such a way that the simulation stage streams out the partial results of all simulation trajectories to the analysis stage that immediately produces a partial result. The simulation-analysis workflow is validated for performance and effectiveness of the online analysis in capturing biological systems behavior on a multicore platform and representative proof-of-concept biological systems. The exploited methodologies include pattern-based parallel programming and data streaming that provide key features to the software designers such as performance portability and efficient in-memory (big) data management and movement. Two paradigmatic classes of biological systems exhibiting multistable and oscillatory behavior are used as a testbed.
Marco Aldinucci, Maurizio Drocco, Guilherme Peretti Pezzi, Claudia Misale, Fabio Tordini, Massimo Torquati
Exercising high-level parallel programming on streams: a systems biology use case Proceedings Article
In: Proc. of 34th IEEE Intl. Conference on Distributed Computing Systems Workshops (ICDCSW), IEEE, Madrid, Spain, 2014.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, impact, paraphrase
@inproceedings{cwc:gpu:dcperf:14,
title = {Exercising high-level parallel programming on streams: a systems biology use case},
author = {Marco Aldinucci and Maurizio Drocco and Guilherme Peretti Pezzi and Claudia Misale and Fabio Tordini and Massimo Torquati},
url = {https://iris.unito.it/retrieve/handle/2318/154516/26657/2014_dcperf_cwc_gpu.pdf},
doi = {10.1109/ICDCSW.2014.38},
year = {2014},
date = {2014-01-01},
booktitle = {Proc. of 34th IEEE Intl. Conference on Distributed Computing Systems Workshops (ICDCSW)},
publisher = {IEEE},
address = {Madrid, Spain},
abstract = {The stochastic modelling of biological systems, cou- pled with Monte Carlo simulation of models, is an increasingly popular technique in Bioinformatics. The simulation-analysis workflow may result into a computationally expensive task reducing the interactivity required in the model tuning. In this work, we advocate high-level software design as a vehicle for building efficient and portable parallel simulators for a variety of platforms, ranging from multi-core platforms to GPGPUs to cloud. In particular, the Calculus of Wrapped Compartments (CWC) parallel simulator for systems biology equipped with on- line mining of results, which is designed according to the FastFlow pattern-based approach, is discussed as a running example. In this work, the CWC simulator is used as a paradigmatic example of a complex C++ application where the quality of results is correlated with both computation and I/O bounds, and where high-quality results might turn into big data. The FastFlow parallel programming framework, which advocates C++ pattern- based parallel programming makes it possible to develop portable parallel code without relinquish neither run-time efficiency nor performance tuning opportunities. Performance and effectiveness of the approach are validated on a variety of platforms, inter-alia cache-coherent multi-cores, cluster of multi-core (Ethernet and Infiniband) and the Amazon Elastic Compute Cloud.},
keywords = {bioinformatics, fastflow, impact, paraphrase},
pubstate = {published},
tppubtype = {inproceedings}
}
The stochastic modelling of biological systems, cou- pled with Monte Carlo simulation of models, is an increasingly popular technique in Bioinformatics. The simulation-analysis workflow may result into a computationally expensive task reducing the interactivity required in the model tuning. In this work, we advocate high-level software design as a vehicle for building efficient and portable parallel simulators for a variety of platforms, ranging from multi-core platforms to GPGPUs to cloud. In particular, the Calculus of Wrapped Compartments (CWC) parallel simulator for systems biology equipped with on- line mining of results, which is designed according to the FastFlow pattern-based approach, is discussed as a running example. In this work, the CWC simulator is used as a paradigmatic example of a complex C++ application where the quality of results is correlated with both computation and I/O bounds, and where high-quality results might turn into big data. The FastFlow parallel programming framework, which advocates C++ pattern- based parallel programming makes it possible to develop portable parallel code without relinquish neither run-time efficiency nor performance tuning opportunities. Performance and effectiveness of the approach are validated on a variety of platforms, inter-alia cache-coherent multi-cores, cluster of multi-core (Ethernet and Infiniband) and the Amazon Elastic Compute Cloud.
Marco Aldinucci, Guilherme Peretti Pezzi, Maurizio Drocco, Fabio Tordini, Peter Kilpatrick, Massimo Torquati
Parallel video denoising on heterogeneous platforms Proceedings Article
In: Proc. of Intl. Workshop on High-level Programming for Heterogeneous and Hierarchical Parallel Systems (HLPGPU), 2014.
Abstract | Links | BibTeX | Tags: fastflow, impact, paraphrase
@inproceedings{ff:video:hlpgpu:14,
title = {Parallel video denoising on heterogeneous platforms},
author = {Marco Aldinucci and Guilherme Peretti Pezzi and Maurizio Drocco and Fabio Tordini and Peter Kilpatrick and Massimo Torquati},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_ff_video_denoiser_hlpgpu.pdf},
year = {2014},
date = {2014-01-01},
booktitle = {Proc. of Intl. Workshop on High-level Programming for Heterogeneous and Hierarchical Parallel Systems (HLPGPU)},
abstract = {In this paper, a highly-effective parallel filter for video denoising is presented. The filter is designed using a skeletal approach, and has been implemented by way of the FastFlow parallel programming library. As a result of its high-level design, it is possible to run the filter seamlessly on a multi-core machine, on GPGPU(s), or on both. The design and the implementation of the filter are discussed, and an experimental evaluation is presented. Various mappings of the filtering stages are comparatively discussed.},
keywords = {fastflow, impact, paraphrase},
pubstate = {published},
tppubtype = {inproceedings}
}
In this paper, a highly-effective parallel filter for video denoising is presented. The filter is designed using a skeletal approach, and has been implemented by way of the FastFlow parallel programming library. As a result of its high-level design, it is possible to run the filter seamlessly on a multi-core machine, on GPGPU(s), or on both. The design and the implementation of the filter are discussed, and an experimental evaluation is presented. Various mappings of the filtering stages are comparatively discussed.
Claudia Misale
Accelerating Bowtie2 with a lock-less concurrency approach and memory affinity Proceedings Article
In: Aldinucci, Marco, D'Agostino, Daniele, Kilpatrick, Peter (Ed.): Proc. of Intl. Euromicro PDP 2014: Parallel Distributed and network-based Processing, IEEE, Torino, Italy, 2014, ((Best paper award)).
Abstract | Links | BibTeX | Tags: fastflow, paraphrase
@inproceedings{ff:bowtie2:pdp:14,
title = {Accelerating Bowtie2 with a lock-less concurrency approach and memory affinity},
author = {Claudia Misale},
editor = {Marco Aldinucci and Daniele D'Agostino and Peter Kilpatrick},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_pdp_bowtieff.pdf},
doi = {10.1109/PDP.2014.50},
year = {2014},
date = {2014-01-01},
booktitle = {Proc. of Intl. Euromicro PDP 2014: Parallel Distributed and network-based Processing},
publisher = {IEEE},
address = {Torino, Italy},
abstract = {The implementation of DNA alignment tools for Bioinformatics lead to face different problems that dip into performances. A single alignment takes an amount of time that is not predictable and there are different factors that can affect performances, for instance the length of sequences can determine the computational grain of the task and mismatches or insertion/deletion (indels) increase time needed to complete an alignment. Moreover, an alignment is a strong memory- bound problem because of the irregular memory access pat- terns and limitations in memory-bandwidth. Over the years, many alignment tools were implemented. A concrete example is Bowtie2, one of the fastest (concurrent, Pthread-based) and state of the art not GPU-based alignment tool. Bowtie2 exploits concurrency by instantiating a pool of threads, which have access to a global input dataset, share the reference genome and have access to different objects for collecting alignment results. In this paper a modified implementation of Bowtie2 is presented, in which the concurrency structure has been changed. The proposed implementation exploits the task-farm skeleton pattern implemented as a Master-Worker. The Master-Worker pattern permits to delegate only to the Master thread dataset reading and to make private to each Worker data structures that are shared in the original version. Only the reference genome is left shared. As a further optimisation, the Master and each Worker were pinned on cores and the reference genome was allocated interleaved among memory nodes. The proposed implementation is able to gain up to 10 speedup points over the original implementation.},
note = {(Best paper award)},
keywords = {fastflow, paraphrase},
pubstate = {published},
tppubtype = {inproceedings}
}
The implementation of DNA alignment tools for Bioinformatics lead to face different problems that dip into performances. A single alignment takes an amount of time that is not predictable and there are different factors that can affect performances, for instance the length of sequences can determine the computational grain of the task and mismatches or insertion/deletion (indels) increase time needed to complete an alignment. Moreover, an alignment is a strong memory- bound problem because of the irregular memory access pat- terns and limitations in memory-bandwidth. Over the years, many alignment tools were implemented. A concrete example is Bowtie2, one of the fastest (concurrent, Pthread-based) and state of the art not GPU-based alignment tool. Bowtie2 exploits concurrency by instantiating a pool of threads, which have access to a global input dataset, share the reference genome and have access to different objects for collecting alignment results. In this paper a modified implementation of Bowtie2 is presented, in which the concurrency structure has been changed. The proposed implementation exploits the task-farm skeleton pattern implemented as a Master-Worker. The Master-Worker pattern permits to delegate only to the Master thread dataset reading and to make private to each Worker data structures that are shared in the original version. Only the reference genome is left shared. As a further optimisation, the Master and each Worker were pinned on cores and the reference genome was allocated interleaved among memory nodes. The proposed implementation is able to gain up to 10 speedup points over the original implementation.
Alessandro Secco, Irfan Uddin, Guilherme Peretti Pezzi, Massimo Torquati
Message passing on InfiniBand RDMA for parallel run-time supports Proceedings Article
In: Aldinucci, Marco, D'Agostino, Daniele, Kilpatrick, Peter (Ed.): Proc. of Intl. Euromicro PDP 2014: Parallel Distributed and network-based Processing, IEEE, Torino, Italy, 2014.
Abstract | Links | BibTeX | Tags: fastflow, impact, paraphrase
@inproceedings{ff:infiniband:pdp:14,
title = {Message passing on InfiniBand RDMA for parallel run-time supports},
author = {Alessandro Secco and Irfan Uddin and Guilherme Peretti Pezzi and Massimo Torquati},
editor = {Marco Aldinucci and Daniele D'Agostino and Peter Kilpatrick},
url = {https://iris.unito.it/retrieve/handle/2318/151178/690885/2014_ff_infiniband_pdp.pdf},
doi = {10.1109/PDP.2014.23},
year = {2014},
date = {2014-01-01},
booktitle = {Proc. of Intl. Euromicro PDP 2014: Parallel Distributed and network-based Processing},
publisher = {IEEE},
address = {Torino, Italy},
abstract = {InfiniBand networks are commonly used in the high performance computing area. They offer RDMA-based opera- tions that help to improve the performance of communication subsystems. In this paper, we propose a minimal message-passing communication layer providing the programmer with a point-to- point communication channel implemented by way of InfiniBand RDMA features. Differently from other libraries exploiting the InfiniBand features, such as the well-known Message Passing Interface (MPI), the proposed library is a communication layer only rather than a programming model, and can be easily used as building block for high-level parallel programming frameworks. Evaluated on micro-benchmarks, the proposed RDMA-based communication channel implementation achieves a comparable performance with highly optimised MPI/InfiniBand implemen- tations. Eventually, the flexibility of the communication layer is evaluated by integrating it within the FastFlow parallel frame- work, currently supporting TCP/IP networks (via the ZeroMQ communication library).},
keywords = {fastflow, impact, paraphrase},
pubstate = {published},
tppubtype = {inproceedings}
}
InfiniBand networks are commonly used in the high performance computing area. They offer RDMA-based opera- tions that help to improve the performance of communication subsystems. In this paper, we propose a minimal message-passing communication layer providing the programmer with a point-to- point communication channel implemented by way of InfiniBand RDMA features. Differently from other libraries exploiting the InfiniBand features, such as the well-known Message Passing Interface (MPI), the proposed library is a communication layer only rather than a programming model, and can be easily used as building block for high-level parallel programming frameworks. Evaluated on micro-benchmarks, the proposed RDMA-based communication channel implementation achieves a comparable performance with highly optimised MPI/InfiniBand implemen- tations. Eventually, the flexibility of the communication layer is evaluated by integrating it within the FastFlow parallel frame- work, currently supporting TCP/IP networks (via the ZeroMQ communication library).
Maurizio Drocco, Marco Aldinucci, Massimo Torquati
A Dynamic Memory Allocator for heterogeneous platforms Proceedings Article
In: Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) – Poster Abstracts, HiPEAC, Fiuggi, Italy, 2014.
Abstract | Links | BibTeX | Tags: fastflow, HPC
@inproceedings{ff:acaces:14,
title = {A Dynamic Memory Allocator for heterogeneous platforms},
author = {Maurizio Drocco and Marco Aldinucci and Massimo Torquati},
url = {http://calvados.di.unipi.it/storage/paper_files/2014_ACACES_ex-abstract.pdf},
year = {2014},
date = {2014-01-01},
booktitle = {Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) – Poster Abstracts},
publisher = {HiPEAC},
address = {Fiuggi, Italy},
abstract = {Modern computers are built upon heterogeneous multi-core/many cores architectures (e.g. GPGPU connected to multi-core CPU). Achieving peak performance on these architectures is hard and may require a substantial programming effort. High-level programming patterns, coupled with efficient low-level runtime supports, have been proposed to relieve the programmer from worrying about low-level details such as synchronisation of racing processes as well as those fine tunings needed to improve the overall performance. Among them are (parallel) dynamic memory allocation and effective exploitation of the memory hierarchy. The memory allocator is often a bottleneck that severely limits program scalability, robustness and portability on parallel systems. In this work we introduce a novel memory allocator, based on the FastFlow's allocator and the recently proposed CUDA Unified Memory, which aims to efficiently integrate host and device memories into a unique dynamic-allocable memory space, accessible transparently by both host and device code.},
keywords = {fastflow, HPC},
pubstate = {published},
tppubtype = {inproceedings}
}
Modern computers are built upon heterogeneous multi-core/many cores architectures (e.g. GPGPU connected to multi-core CPU). Achieving peak performance on these architectures is hard and may require a substantial programming effort. High-level programming patterns, coupled with efficient low-level runtime supports, have been proposed to relieve the programmer from worrying about low-level details such as synchronisation of racing processes as well as those fine tunings needed to improve the overall performance. Among them are (parallel) dynamic memory allocation and effective exploitation of the memory hierarchy. The memory allocator is often a bottleneck that severely limits program scalability, robustness and portability on parallel systems. In this work we introduce a novel memory allocator, based on the FastFlow's allocator and the recently proposed CUDA Unified Memory, which aims to efficiently integrate host and device memories into a unique dynamic-allocable memory space, accessible transparently by both host and device code.
Claudia Misale, Giulio Ferrero, Massimo Torquati, Marco Aldinucci
Sequence alignment tools: one parallel pattern to rule them all? Journal Article
In: BioMed Research International, 2014.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow, paraphrase, repara
@article{bowtie-bwa:ff:multicore:biomed:14,
title = {Sequence alignment tools: one parallel pattern to rule them all?},
author = {Claudia Misale and Giulio Ferrero and Massimo Torquati and Marco Aldinucci},
url = {http://downloads.hindawi.com/journals/bmri/2014/539410.pdf},
doi = {10.1155/2014/539410},
year = {2014},
date = {2014-01-01},
journal = {BioMed Research International},
abstract = {In this paper we advocate high-level programming methodology for Next Generation Sequencers (NGS) alignment tools for both productivity and absolute performance. We analyse the problem of parallel alignment and review the parallelisation strategies of the most popular alignment tools, which can all be abstracted to a single parallel paradigm. We compare these tools against their porting onto the FastFlow pattern-based programming framework, which provides programmers with high-level parallel patterns. By using a high-level approach, programmers are liberated from all complex aspects of parallel programming, such as synchronisation protocols and task scheduling, gaining more possibility for seamless performance tuning. In this work we show some use case in which, by using a high-level approach for parallelising NGS tools, it is possible to obtain comparable or even better absolute performance for all used datasets.},
keywords = {bioinformatics, fastflow, paraphrase, repara},
pubstate = {published},
tppubtype = {article}
}
In this paper we advocate high-level programming methodology for Next Generation Sequencers (NGS) alignment tools for both productivity and absolute performance. We analyse the problem of parallel alignment and review the parallelisation strategies of the most popular alignment tools, which can all be abstracted to a single parallel paradigm. We compare these tools against their porting onto the FastFlow pattern-based programming framework, which provides programmers with high-level parallel patterns. By using a high-level approach, programmers are liberated from all complex aspects of parallel programming, such as synchronisation protocols and task scheduling, gaining more possibility for seamless performance tuning. In this work we show some use case in which, by using a high-level approach for parallelising NGS tools, it is possible to obtain comparable or even better absolute performance for all used datasets.
2013
Maurizio Drocco
Parallel stochastic simulators in systems biology: the evolution of the species Masters Thesis
Computer Science Department, University of Torino, Italy, 2013.
Abstract | Links | BibTeX | Tags: fastflow
@mastersthesis{tesi:drocco:13,
title = {Parallel stochastic simulators in systems biology: the evolution of the species},
author = {Maurizio Drocco},
url = {http://calvados.di.unipi.it/storage/paper_files/2013_tesi_drocco.pdf},
year = {2013},
date = {2013-07-01},
school = {Computer Science Department, University of Torino, Italy},
abstract = {The stochastic simulation of biological systems is an increasingly popular technique in bioinformatics. It is often an enlightening technique, especially for multi-stable systems whose dynamics can be hardly captured with ordinary differential equations. To be effective, stochastic simulations should be supported by powerful statistical analysis tools. The simulation/analysis workflow may however result in being computationally expensive, thus compromising the interactivity required especially in model tuning. In this work we discuss the main opportunities to speed up the framework by parallelisation on modern multicore and hybrid multicore and distributed platforms, advocating the high-level design of simulators for stochastic systems as a vehicle for building efficient and portable parallel simulators endowed with on-line statistical analysis. In particular, the Calculus of Wrapped Compartments (CWC) Simulator, which is designed according to the FastFlow's pattern-based approach, is presented and discussed in this work.},
keywords = {fastflow},
pubstate = {published},
tppubtype = {mastersthesis}
}
The stochastic simulation of biological systems is an increasingly popular technique in bioinformatics. It is often an enlightening technique, especially for multi-stable systems whose dynamics can be hardly captured with ordinary differential equations. To be effective, stochastic simulations should be supported by powerful statistical analysis tools. The simulation/analysis workflow may however result in being computationally expensive, thus compromising the interactivity required especially in model tuning. In this work we discuss the main opportunities to speed up the framework by parallelisation on modern multicore and hybrid multicore and distributed platforms, advocating the high-level design of simulators for stochastic systems as a vehicle for building efficient and portable parallel simulators endowed with on-line statistical analysis. In particular, the Calculus of Wrapped Compartments (CWC) Simulator, which is designed according to the FastFlow's pattern-based approach, is presented and discussed in this work.
Marco Aldinucci, Fabio Tordini, Maurizio Drocco, Massimo Torquati, Mario Coppo
Parallel stochastic simulators in system biology: the evolution of the species Proceedings Article
In: Proc. of 21st Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP), IEEE, Belfast, Nothern Ireland, U.K., 2013.
Abstract | Links | BibTeX | Tags: bioinformatics, fastflow
@inproceedings{ff_cwc_distr:pdp:13,
title = {Parallel stochastic simulators in system biology: the evolution of the species},
author = {Marco Aldinucci and Fabio Tordini and Maurizio Drocco and Massimo Torquati and Mario Coppo},
url = {http://calvados.di.unipi.it/storage/paper_files/2013_cwc_d_PDP.pdf},
doi = {10.1109/PDP.2013.66},
year = {2013},
date = {2013-02-01},
booktitle = {Proc. of 21st Euromicro Intl. Conference on Parallel Distributed and network-based Processing (PDP)},
publisher = {IEEE},
address = {Belfast, Nothern Ireland, U.K.},
abstract = {The stochastic simulation of biological systems is an increasingly popular technique in Bioinformatics. It is often an enlightening technique, especially for multi-stable systems which dynamics can be hardly captured with ordinary differential equations. To be effective, stochastic simulations should be supported by powerful statistical analysis tools. The simulation-analysis workflow may however result in being computationally expensive, thus compromising the interactivity required in model tuning. In this work we advocate the high-level design of simulators for stochastic systems as a vehicle for building efficient and portable parallel simulators. In particular, the Calculus of Wrapped Components (CWC) simulator, which is designed according to the FastFlow's pattern-based approach, is presented and discussed in this work. FastFlow has been extended to support also clusters of multi-cores with minimal coding effort, assessing the portability of the approach.},
keywords = {bioinformatics, fastflow},
pubstate = {published},
tppubtype = {inproceedings}
}
The stochastic simulation of biological systems is an increasingly popular technique in Bioinformatics. It is often an enlightening technique, especially for multi-stable systems which dynamics can be hardly captured with ordinary differential equations. To be effective, stochastic simulations should be supported by powerful statistical analysis tools. The simulation-analysis workflow may however result in being computationally expensive, thus compromising the interactivity required in model tuning. In this work we advocate the high-level design of simulators for stochastic systems as a vehicle for building efficient and portable parallel simulators. In particular, the Calculus of Wrapped Components (CWC) simulator, which is designed according to the FastFlow's pattern-based approach, is presented and discussed in this work. FastFlow has been extended to support also clusters of multi-cores with minimal coding effort, assessing the portability of the approach.
Claudia Misale, Marco Aldinucci, Massimo Torquati
Memory affinity in multi-threading: the Bowtie2 case study Proceedings Article
In: Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) – Poster Abstracts, HiPEAC, Fiuggi, Italy, 2013, ISBN: 9789038221908.
Abstract | Links | BibTeX | Tags: fastflow
@inproceedings{ff:acaces:13,
title = {Memory affinity in multi-threading: the Bowtie2 case study},
author = {Claudia Misale and Marco Aldinucci and Massimo Torquati},
url = {https://iris.unito.it/retrieve/handle/2318/143005/23874/2013_ACACES_ex-abstract.pdf},
isbn = {9789038221908},
year = {2013},
date = {2013-01-01},
booktitle = {Advanced Computer Architecture and Compilation for High-Performance and Embedded Systems (ACACES) – Poster Abstracts},
publisher = {HiPEAC},
address = {Fiuggi, Italy},
abstract = {The diffusion of the Next Generation Sequencing (NGS) has increased the amount of data obtainable by genomic experiments. From a DNA sample a NGS run is able to produce millions of short sequences (called reads), which should be mapped into a reference genome. In this paper, we analyse the performance of Bowtie2, a fast and popular DNA mapping tool. Bowtie2 exhibits a multithreading implementation on top of pthreads, spin-locks and SSE2 SIMD extension. From parallel computing viewpoint, is a paradigmatic example of a software requiring to address three fundamental problems in shared-memory programming for cache-coherent multi-core platforms: synchronisation efficiency at very fine grain (due to short reads), load-balancing (due to long reads), and efficient usage of memory subsystem (due to SSE2 memory pressure). We compare the original implementation against an alternative implementation on top of the FastFlow pattern-based programming framework. The proposed design exploits the high-level farm pattern of FastFlow, which is implemented top of nonblocking multi-threading and lock-less (CAS-free) queues, and provides the programmer with high-level mechanism to tune task scheduling to achieve both load-balancing and memory affinity. The proposed design, despite the high-level design, is always faster and more scalable with respect to the original one. The design of both original and alternative version will be presented along with their experimental evaluation on real-world data sets.},
keywords = {fastflow},
pubstate = {published},
tppubtype = {inproceedings}
}
The diffusion of the Next Generation Sequencing (NGS) has increased the amount of data obtainable by genomic experiments. From a DNA sample a NGS run is able to produce millions of short sequences (called reads), which should be mapped into a reference genome. In this paper, we analyse the performance of Bowtie2, a fast and popular DNA mapping tool. Bowtie2 exhibits a multithreading implementation on top of pthreads, spin-locks and SSE2 SIMD extension. From parallel computing viewpoint, is a paradigmatic example of a software requiring to address three fundamental problems in shared-memory programming for cache-coherent multi-core platforms: synchronisation efficiency at very fine grain (due to short reads), load-balancing (due to long reads), and efficient usage of memory subsystem (due to SSE2 memory pressure). We compare the original implementation against an alternative implementation on top of the FastFlow pattern-based programming framework. The proposed design exploits the high-level farm pattern of FastFlow, which is implemented top of nonblocking multi-threading and lock-less (CAS-free) queues, and provides the programmer with high-level mechanism to tune task scheduling to achieve both load-balancing and memory affinity. The proposed design, despite the high-level design, is always faster and more scalable with respect to the original one. The design of both original and alternative version will be presented along with their experimental evaluation on real-world data sets.
Marco Aldinucci, Sonia Campa, Peter Kilpatrick, Massimo Torquati
Structured Data Access Annotations for Massively Parallel Computations Proceedings Article
In: Euro-Par 2012 Workshops, Proc. of the ParaPhrase Workshop on Parallel Processing, pp. 381–390, Springer, 2013.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase
@inproceedings{annotation:para:12,
title = {Structured Data Access Annotations for Massively Parallel Computations},
author = {Marco Aldinucci and Sonia Campa and Peter Kilpatrick and Massimo Torquati},
url = {http://calvados.di.unipi.it/storage/paper_files/2013_annot_europar_workshops.pdf},
doi = {10.1007/978-3-642-36949-0_42},
year = {2013},
date = {2013-01-01},
booktitle = {Euro-Par 2012 Workshops, Proc. of the ParaPhrase Workshop on Parallel Processing},
volume = {7640},
pages = {381–390},
publisher = {Springer},
series = {LNCS},
abstract = {We describe an approach aimed at addressing the issue of joint exploitation of control (stream) and data parallelism in a skele-ton based parallel programming environment, based on annotations and refactoring. Annotations drive efficient implementation of a parallel com-putation. Refactoring is used to transform the associated skeleton tree into a more efficient, functionally equivalent skeleton tree. In most cases,cost models are used to drive the refactoring process. We show howsample use case applications/kernels may be optimized and discuss pre-liminary experiments with FastFlow assessing the theoretical results.},
keywords = {fastflow, paraphrase},
pubstate = {published},
tppubtype = {inproceedings}
}
We describe an approach aimed at addressing the issue of joint exploitation of control (stream) and data parallelism in a skele-ton based parallel programming environment, based on annotations and refactoring. Annotations drive efficient implementation of a parallel com-putation. Refactoring is used to transform the associated skeleton tree into a more efficient, functionally equivalent skeleton tree. In most cases,cost models are used to drive the refactoring process. We show howsample use case applications/kernels may be optimized and discuss pre-liminary experiments with FastFlow assessing the theoretical results.
Marco Aldinucci, Sonia Campa, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
Targeting Distributed Systems in FastFlow Proceedings Article
In: Euro-Par 2012 Workshops, Proc. of the CoreGrid Workshop on Grids, Clouds and P2P Computing, pp. 47–56, Springer, 2013.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase
@inproceedings{ff:distr:cgs:12,
title = {Targeting Distributed Systems in FastFlow},
author = {Marco Aldinucci and Sonia Campa and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_distr_ff_cgsymph.pdf},
doi = {10.1007/978-3-642-36949-0_7},
year = {2013},
date = {2013-01-01},
booktitle = {Euro-Par 2012 Workshops, Proc. of the CoreGrid Workshop on Grids, Clouds and P2P Computing},
volume = {7640},
pages = {47–56},
publisher = {Springer},
series = {LNCS},
abstract = {FastFlow is a structured parallel programming framework targeting shared memory multi-core architectures. In this paper we introduce a FastFlow extension aimed at supporting a network of multi-core workstation as well. The extension supports the execution of FastFlow programs by coordinating – in a structured way – the fine grain parallel activities running on a single workstation. We discuss the design and the implementation of this extension presenting preliminary experimental results validating it on state-of-the-art networked multi-core nodes.},
keywords = {fastflow, paraphrase},
pubstate = {published},
tppubtype = {inproceedings}
}
FastFlow is a structured parallel programming framework targeting shared memory multi-core architectures. In this paper we introduce a FastFlow extension aimed at supporting a network of multi-core workstation as well. The extension supports the execution of FastFlow programs by coordinating – in a structured way – the fine grain parallel activities running on a single workstation. We discuss the design and the implementation of this extension presenting preliminary experimental results validating it on state-of-the-art networked multi-core nodes.
Marco Aldinucci, Sonia Campa, Fabio Tordini, Massimo Torquati, Peter Kilpatrick
An abstract annotation model for skeletons Book Section
In: Beckert, Bernhard, Damiani, Ferruccio, Boer, Frank S., Bonsangue, Marcello M. (Ed.): Formal Methods for Components and Objects: Intl. Symposium, FMCO 2011, Torino, Italy, October 3-5, 2011, Revised Invited Lectures, vol. 7542, pp. 257–276, Springer, 2013, ISBN: 978-3-642-35886-9.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase
@incollection{toolchain:fmco:11,
title = {An abstract annotation model for skeletons},
author = {Marco Aldinucci and Sonia Campa and Fabio Tordini and Massimo Torquati and Peter Kilpatrick},
editor = {Bernhard Beckert and Ferruccio Damiani and Frank S. Boer and Marcello M. Bonsangue},
url = {http://calvados.di.unipi.it/storage/paper_files/2013_fmco11_annotation.pdf},
doi = {10.1007/978-3-642-35887-6_14},
isbn = {978-3-642-35886-9},
year = {2013},
date = {2013-01-01},
booktitle = {Formal Methods for Components and Objects: Intl. Symposium, FMCO 2011, Torino, Italy, October 3-5, 2011, Revised Invited Lectures},
volume = {7542},
pages = {257–276},
publisher = {Springer},
series = {LNCS},
abstract = {Multi-core and many-core platforms are becoming increasingly heterogeneous and asymmetric. This significantly increases the porting and tuning effort required for parallel codes, which in turn often leads to a growing gap between peak machine power and actual application performance. In this work a first step toward the automated optimization of high level skeleton-based parallel code is discussed. The paper presents an abstract annotation model for skeleton programs aimed at formally describing suitable mapping of parallel activities on a high-level platform representation. The derived mapping and scheduling strategies are used to generate optimized run-time code.},
keywords = {fastflow, paraphrase},
pubstate = {published},
tppubtype = {incollection}
}
Multi-core and many-core platforms are becoming increasingly heterogeneous and asymmetric. This significantly increases the porting and tuning effort required for parallel codes, which in turn often leads to a growing gap between peak machine power and actual application performance. In this work a first step toward the automated optimization of high level skeleton-based parallel code is discussed. The paper presents an abstract annotation model for skeleton programs aimed at formally describing suitable mapping of parallel activities on a high-level platform representation. The derived mapping and scheduling strategies are used to generate optimized run-time code.
2012
Marco Aldinucci, Concetto Spampinato, Maurizio Drocco, Massimo Torquati, Simone Palazzo
A Parallel Edge Preserving Algorithm for Salt and Pepper Image Denoising Proceedings Article
In: Djemal, K., Deriche, M., Puech, W., Ucan, Osman N. (Ed.): Proc. of 2nd Intl. Conference on Image Processing Theory Tools and Applications (IPTA), pp. 97–102, IEEE, Istambul, Turkey, 2012, ISBN: 978-1-4673-2582-0.
Abstract | Links | BibTeX | Tags: fastflow, impact
@inproceedings{denoiser:ff:ipta:12,
title = {A Parallel Edge Preserving Algorithm for Salt and Pepper Image Denoising},
author = {Marco Aldinucci and Concetto Spampinato and Maurizio Drocco and Massimo Torquati and Simone Palazzo},
editor = {K. Djemal and M. Deriche and W. Puech and Osman N. Ucan},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_2phasedenoiser_ff_ipta.pdf},
doi = {10.1109/IPTA.2012.6469567},
isbn = {978-1-4673-2582-0},
year = {2012},
date = {2012-10-01},
booktitle = {Proc. of 2nd Intl. Conference on Image Processing Theory Tools and Applications (IPTA)},
pages = {97–102},
publisher = {IEEE},
address = {Istambul, Turkey},
abstract = {In this paper a two-phase filter for removing ``salt and pepper'' noise is proposed. In the first phase, an adaptive median filter is used to identify the set of the noisy pixels; in the second phase, these pixels are restored according to a regularization method, which contains a data-fidelity term reflecting the impulse noise characteristics. The algorithm, which exhibits good performance both in denoising and in restoration, can be easily and effectively parallelized to exploit the full power of multi-core CPUs and GPGPUs; the proposed implementation based on the FastFlow library achieves both close-to-ideal speedup and very good wall-clock execution figures.},
keywords = {fastflow, impact},
pubstate = {published},
tppubtype = {inproceedings}
}
In this paper a two-phase filter for removing ``salt and pepper'' noise is proposed. In the first phase, an adaptive median filter is used to identify the set of the noisy pixels; in the second phase, these pixels are restored according to a regularization method, which contains a data-fidelity term reflecting the impulse noise characteristics. The algorithm, which exhibits good performance both in denoising and in restoration, can be easily and effectively parallelized to exploit the full power of multi-core CPUs and GPGPUs; the proposed implementation based on the FastFlow library achieves both close-to-ideal speedup and very good wall-clock execution figures.
Marco Aldinucci, Marco Danelutto, Peter Kilpatrick, Massimiliano Meneghin, Massimo Torquati
An Efficient Unbounded Lock-Free Queue for Multi-core Systems Proceedings Article
In: Proc. of 18th Intl. Euro-Par 2012 Parallel Processing, pp. 662–673, Springer, Rhodes Island, Greece, 2012.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase
@inproceedings{ff:spsc:europar:12,
title = {An Efficient Unbounded Lock-Free Queue for Multi-core Systems},
author = {Marco Aldinucci and Marco Danelutto and Peter Kilpatrick and Massimiliano Meneghin and Massimo Torquati},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_spsc_europar.pdf},
doi = {10.1007/978-3-642-32820-6_65},
year = {2012},
date = {2012-08-01},
booktitle = {Proc. of 18th Intl. Euro-Par 2012 Parallel Processing},
volume = {7484},
pages = {662–673},
publisher = {Springer},
address = {Rhodes Island, Greece},
series = {LNCS},
abstract = {The use of efficient synchronization mechanisms is crucial for implementing fine grained parallel programs on modern shared cache multi-core architectures. In this paper we study this problem by considering Single-Producer/Single-Consumer (SPSC) coordination using unbounded queues. A novel unbounded SPSC algorithm capable of reducing the row synchronization latency and speeding up Producer-Consumer coordination is presented. The algorithm has been extensively tested on a shared-cache multi-core platform and a sketch proof of correctness is presented. The queues proposed have been used as basic building blocks to implement the FastFlow parallel framework, which has been demonstrated to offer very good performance for fine-grain parallel applications.},
keywords = {fastflow, paraphrase},
pubstate = {published},
tppubtype = {inproceedings}
}
The use of efficient synchronization mechanisms is crucial for implementing fine grained parallel programs on modern shared cache multi-core architectures. In this paper we study this problem by considering Single-Producer/Single-Consumer (SPSC) coordination using unbounded queues. A novel unbounded SPSC algorithm capable of reducing the row synchronization latency and speeding up Producer-Consumer coordination is presented. The algorithm has been extensively tested on a shared-cache multi-core platform and a sketch proof of correctness is presented. The queues proposed have been used as basic building blocks to implement the FastFlow parallel framework, which has been demonstrated to offer very good performance for fine-grain parallel applications.
Marco Aldinucci, Marco Danelutto, Peter Kilpatrick, Massimo Torquati
Targeting heterogeneous architectures via macro data flow Journal Article
In: Parallel Processing Letters, vol. 22, no. 2, 2012, ISSN: 0129-6264.
Abstract | Links | BibTeX | Tags: fastflow, paraphrase
@article{mdf:hplgpu:ppl:12,
title = {Targeting heterogeneous architectures via macro data flow},
author = {Marco Aldinucci and Marco Danelutto and Peter Kilpatrick and Massimo Torquati},
url = {http://calvados.di.unipi.it/storage/paper_files/2012_mdf_PPL-hplgpu.pdf},
doi = {10.1142/S0129626412400063},
issn = {0129-6264},
year = {2012},
date = {2012-06-01},
journal = {Parallel Processing Letters},
volume = {22},
number = {2},
abstract = {We propose a data flow based run time system as an efficient tool for supporting execution of parallel code on heterogeneous architectures hosting both multicore CPUs and GPUs. We discuss how the proposed run time system may be the target of both structured parallel applications developed using algorithmic skeletons/parallel design patterns and also more ``domain specific'' programming models. Experimental results demonstrating the feasibility of the approach are presented.},
keywords = {fastflow, paraphrase},
pubstate = {published},
tppubtype = {article}
}
We propose a data flow based run time system as an efficient tool for supporting execution of parallel code on heterogeneous architectures hosting both multicore CPUs and GPUs. We discuss how the proposed run time system may be the target of both structured parallel applications developed using algorithmic skeletons/parallel design patterns and also more ``domain specific'' programming models. Experimental results demonstrating the feasibility of the approach are presented.
Talks
Sorry, no publications matched your criteria.
Discontinued Parallel Computing tools
Read more
GAM: Global Asynchronous Memory (2018)
Parallel Programming with Global Asynchronous Memory: Models, C++ APIs and Implementations
M. Drocco, “Parallel programming with global asynchronous memory: models, C++ APIs and implementations,” PhD Thesis, 2017. doi:10.5281/zenodo.1037585
PiCo: Pipeline Composition (2017)
PiCo (Pipeline Composition) is an open-source C++11 header-only DSL for high-performance data analytics, featuring low latency, high throughput, and minimal memory footprint on multi-core platforms. For more information see the PiCo paper.
GridCOMP Grid Component Model (2006)
The full software package supporting the development of distributed and multi-core applications based on autonomic components and behavioural skeletons is available under GPL license. More information on the GridCOMP page. The Grid Component Model (GCM) has been standardised by ETSI: DTS/GRID-0004-1 (27/08/2008), DTS/GRID-004-2 (27/08/2008), DTS/GRID-0004-3 (20/03/2009), DTS/GRID-0004-4 (24/03/2010).
VirtuaLinux (2006)
VirtuaLinux is a Linux meta-distribution that allows the creation, deployment and administration of virtualized clusters with no single point of failure. VirtuaLinux architecture supports disk-less configurations and provides an efficient, iSCSI based abstraction of the SAN. Clusters running VirtuaLinux exhibits no master node to boost resilience and flexibility. Thanks to its storage virtualisation layer, VIrtuaLinux was able to deploy hundreds of VMs in a few seconds. Actually, VirtuaLinux realises a cloud (but the cloud word with the current meaning did not exist in 2006).
Muskel (2005)
Muskel is a parallel programming library providing users with structured parallel constructs (skeletons) that can be used to implement efficient parallel applications. Muskel applications run on networks/clusters of workstations equipped with Java (1.5 or greater). The skeletons are implemented exploiting macro data flow technology. Muskel extends Lithium with many interesting features, in particular with adaptive and autonomic features.
Ad-HOC(2004)
AD-HOC (Adaptive Distributed Herd of Object Caches), is a fast and robust distributed object repository. It provides applications with a distributed storage manager that virtualise PC’s memories into a unique common distributed storage space. Ad-HOC can effectively be used to implement DSMs as well as distributed cache subsystems. a high-performance distributed shared memory server for cluster and grid, and its applications. ADHOC is a basic block enabling the development of shared memory run-time supports and applications for dynamic and unreliable executing environments (C++, GPL). The libraries and applications developed on top of ADHOC include:
- a parallel file system exhibiting the same API and better performance of the PVFS;
- a distributed cache that can be plugged in the Apache webserver with no modifications of Apache code. The cache substantially improve web server farm performance with no additional costs;
- a Distributed Shared Memory (DSM) for ASSIST.
ASSIST (2003)
ASSIST (A Software development System based on Integrated Skeleton Technology) is a parallel programming environment based on skeleton and coordination language technology aimed at the development of distributed high-performance applications. ASSIST applications should be compiled in binary packages that can be deployed and run on grids, including those exhibiting heterogeneous platforms. Deployment and run are provided through standard middleware services (e.g. Globus) enriched with the ASSIST run-time support. ASSIST applications are described by means of a coordination language, which can express arbitrary graphs of modules, interconnected by typed streams of data. For more information see ASSIST papers.
Lithium (2002)
Lithium is a Java-based parallel programming library providing users with structured parallel constructs (patterns/skeletons) that can be used to implement efficient parallel applications on clusters. The skeletons (including pipe, farm, map, reduce, loop) are implemented exploiting macro data flow technology. Lithium skeletons admit a formal specification of both functional and extra-functional behaviour.
Eskimo (2002)
Eskimo (Easy SKeleton Interface – Memory Oriented), which was part of my PhD dissertation, is a first (maybe a bit naive) tentative to bring skeletal/pattern-based programming on the shared memory model. To my knowledge, there were no previous experiments since skeletal programming was exclusively living in the message passing arena. From a certain viewpoint, it can be considered an ancestor of Fastflow (and other libraries in this class, such as Intel TBB).
Meta (2000)
META is a toolkit for the source-to-source optimisation of pattern-based/skeletal parallel programs (OCaml, GPL). It includes a quite efficient subtree-matching implementation.
SkIE (1998)
SkIE (Skeleton-based Integrated Environment) is a skeleton-based parallel programming environment. SkIE was an engineered version of P3L developed within Quadrics Supercomputing World (QSW) and Alenia Aerospace. Within QSW, I have designed and developed part of the compiler back-end.