Talks | Parallel Computing
2024
Gianluca Mittone, Alberto Mulone, Iacopo Colonnelli, Robert Birke, Marco Aldinucci
Enabling Cross-Facility LLMs Pre-Training Miscellaneous
Accelerating the Development and Use of Generative AI for Science and Engineering: The Trillion Parameter Consortium (TPC), 2024.
Abstract | Links | BibTeX | Tags: eupilot, fl, icsc, space, streamflow
@misc{24:mittone:TPC,
title = {Enabling Cross-Facility LLMs Pre-Training},
author = {Gianluca Mittone and Alberto Mulone and Iacopo Colonnelli and Robert Birke and Marco Aldinucci},
url = {https://datacloud.di.unito.it/index.php/s/DRgm8ebBkKQgD2d},
year = {2024},
date = {2024-11-01},
address = {Atlanta, GE, USA},
abstract = {Big-tech companies pre-train SOTA LLMs on special-purpose, private HPCs, while public research centres lack the resources to compete. We advocate a new take on large model training, e.g., LLMs, called xFFL, which leverages federated learning as an enabling technique to exploit geographically distributed computing power to bridge such digital divide. This work introduces a proof-of-concept federated training of LLaMA-3 8B on three EuroHPC Top500 facilities, proving the viability of leveraging cross-facility publicly available computational power to sustain SOTA LLM workloads.},
howpublished = {Accelerating the Development and Use of Generative AI for Science and Engineering: The Trillion Parameter Consortium (TPC)},
keywords = {eupilot, fl, icsc, space, streamflow},
pubstate = {published},
tppubtype = {misc}
}
Big-tech companies pre-train SOTA LLMs on special-purpose, private HPCs, while public research centres lack the resources to compete. We advocate a new take on large model training, e.g., LLMs, called xFFL, which leverages federated learning as an enabling technique to exploit geographically distributed computing power to bridge such digital divide. This work introduces a proof-of-concept federated training of LLaMA-3 8B on three EuroHPC Top500 facilities, proving the viability of leveraging cross-facility publicly available computational power to sustain SOTA LLM workloads.
Roberto Esposito Mirko Polato Samuele Fonio
FedHP: Federated Learning with Hyperspherical Prototypical Regularization Miscellaneous
32nd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, (ESANN), 2024.
Abstract | Links | BibTeX | Tags: ai, fl, icsc
@misc{24:esann:fedhp,
title = {FedHP: Federated Learning with Hyperspherical Prototypical Regularization},
author = {Roberto Esposito Mirko Polato Samuele Fonio},
url = {https://datacloud.di.unito.it/index.php/s/fKyKSSFQKT3LTxW},
year = {2024},
date = {2024-10-01},
address = {Bruges, Belgium},
abstract = {This paper introduces FedHP, an innovative algorithm that integrates federated learning, hyperspherical geometries, and prototype learning. Federated Learning (FL) has gained prominence as a privacy- preserving method for building robust models across distributed datasets. Traditionally, FL exchanges model parameters to maintain data privacy; however, in scenarios with expensive data communication, exchanging large neural network models becomes impractical. In such cases, prototype learning offers a viable solution by facilitating the exchange of only a few prototypes. Motivated by these considerations, our approach capitalizes on recent advancements in prototype learning, particularly the advantages offered by non-Euclidean geometries. In addition to presenting FedHP, we offer empirical evidence demonstrating its comparability to other state-of- the-art approaches while significantly reducing communication costs.},
howpublished = {32nd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, (ESANN)},
keywords = {ai, fl, icsc},
pubstate = {published},
tppubtype = {misc}
}
This paper introduces FedHP, an innovative algorithm that integrates federated learning, hyperspherical geometries, and prototype learning. Federated Learning (FL) has gained prominence as a privacy- preserving method for building robust models across distributed datasets. Traditionally, FL exchanges model parameters to maintain data privacy; however, in scenarios with expensive data communication, exchanging large neural network models becomes impractical. In such cases, prototype learning offers a viable solution by facilitating the exchange of only a few prototypes. Motivated by these considerations, our approach capitalizes on recent advancements in prototype learning, particularly the advantages offered by non-Euclidean geometries. In addition to presenting FedHP, we offer empirical evidence demonstrating its comparability to other state-of- the-art approaches while significantly reducing communication costs.
Gianluca Mittone
Pushing Federated Learning Boundaries: Three Innovative Distributed Intelligence Approaches Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: eupilot, fl, icsc
@misc{24:mittone:bighpc,
title = {Pushing Federated Learning Boundaries: Three Innovative Distributed Intelligence Approaches},
author = {Gianluca Mittone},
url = {https://datacloud.di.unito.it/index.php/s/eKbRtSAEdmSFJYW},
year = {2024},
date = {2024-09-01},
address = {Pisa, Italy},
abstract = {Federated learning is a distributed, privacy-preserving machine learning technique used on private, decentralised data. It allows multiple parties to cooperatively solve a common machine learning problem without sharing the local data. Three assumptions of state-of-the-art federated learning software constitute the starting points for this research work: 1) their inner workings being strictly tied to deep learning models, 2) the centralised structure currently implemented by many commercial frameworks, and 3) their assumption of being deployed on private, specialised computing infrastructures. The proposed research expands the federated learning paradigm to handle scenarios in which these three conditions do not hold. Such research problems are addressed methodologically and practically, and three open-source, proof-of-concept software are made freely available as tangible research results: OpenFL-x, FastFL, and xFFL.},
keywords = {eupilot, fl, icsc},
pubstate = {published},
tppubtype = {misc}
}
Federated learning is a distributed, privacy-preserving machine learning technique used on private, decentralised data. It allows multiple parties to cooperatively solve a common machine learning problem without sharing the local data. Three assumptions of state-of-the-art federated learning software constitute the starting points for this research work: 1) their inner workings being strictly tied to deep learning models, 2) the centralised structure currently implemented by many commercial frameworks, and 3) their assumption of being deployed on private, specialised computing infrastructures. The proposed research expands the federated learning paradigm to handle scenarios in which these three conditions do not hold. Such research problems are addressed methodologically and practically, and three open-source, proof-of-concept software are made freely available as tangible research results: OpenFL-x, FastFL, and xFFL.
Samuele Fonio Bruno Casella Oussama Harrak
Federated Adaboost for Survival Analysis Miscellaneous
European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2nd Workshop on Advancements in Federated Learning (WAFL), 2024.
Abstract | Links | BibTeX | Tags: ai, epi, fl, icsc
@misc{23:ecmlpkdd:fedsurvboost,
title = {Federated Adaboost for Survival Analysis},
author = {Samuele Fonio Bruno Casella Oussama Harrak},
url = {https://datacloud.di.unito.it/index.php/s/DtXiQfne6BEC235},
year = {2024},
date = {2024-09-01},
address = {Vilnius, Lithuania},
abstract = {This work proposes FedSurvBoost, a federated learning pipeline for survival analysis based on the AdaBoost.F algorithm, which iteratively aggregates the best local weak hypotheses. Our method extends AdaBoost.F by removing the dependence on the number of classes coefficient from the computation of the weights of the best model. This makes it suitable for regression tasks, such as survival analysis. We show the effectiveness of our approach by comparing it with state-of-the-art methods, specifically developed for survival analysis problems, on two common survival datasets.},
howpublished = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2nd Workshop on Advancements in Federated Learning (WAFL)},
keywords = {ai, epi, fl, icsc},
pubstate = {published},
tppubtype = {misc}
}
This work proposes FedSurvBoost, a federated learning pipeline for survival analysis based on the AdaBoost.F algorithm, which iteratively aggregates the best local weak hypotheses. Our method extends AdaBoost.F by removing the dependence on the number of classes coefficient from the computation of the weights of the best model. This makes it suitable for regression tasks, such as survival analysis. We show the effectiveness of our approach by comparing it with state-of-the-art methods, specifically developed for survival analysis problems, on two common survival datasets.
Gianluca Mittone
Into to Federated Learning Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: fl, icsc
@misc{24:mittone:ictp,
title = {Into to Federated Learning},
author = {Gianluca Mittone},
url = {https://datacloud.di.unito.it/index.php/s/nSwwmedjqe2jbWJ},
year = {2024},
date = {2024-05-01},
address = {Trieste, Italy},
abstract = {Machine Learning (ML) is the branch of Artificial Intelligence focused on developing algorithms capable of adapting and improving their predictive or generative performance by feeding on data. Adapting or improving the system’s behaviour based on the provided data is called learning since it is similar to the human learning process in many aspects. The same ML algorithm, usually referred to as a model, trained on different data will thus expose different capabilities and can, therefore, solve different tasks. FL is a relatively recent distributed ML methodology aiming to bridge the gap between the need to train ever bigger ML models on ever larger datasets and the individual and companies’ will to protect and not share their private data. From another point of view, FL is also a way to distribute the training of an ML model even more than before. However, it should be considered that the learning performance of FL is usually lower than that of traditional centralised learning. This course will start from Kairouz ad McMahan’s definition of FL: ”Federated learning is a machine learning setting where multiple entities (clients) collaborate in solving a machine learning problem, under the coordination of a central server or service provider. Each client’s raw data is stored locally and not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective.” From this starting point, the most significant aspects of FL will be exposed and discussed. This tutorial will particularly explore FL from both the learning and computational [5] performance perspectives, investigating its pros and cons in a distributed ML setting. Since FL natively targets data privacy, some insights on how the FL process can be attacked and protected will also be discussed from a high-level perspective. Finally, a hands-on session will guide the participants in building a basic FL system, providing a better understanding of the major implementational difficulties of such a technique.},
keywords = {fl, icsc},
pubstate = {published},
tppubtype = {misc}
}
Machine Learning (ML) is the branch of Artificial Intelligence focused on developing algorithms capable of adapting and improving their predictive or generative performance by feeding on data. Adapting or improving the system’s behaviour based on the provided data is called learning since it is similar to the human learning process in many aspects. The same ML algorithm, usually referred to as a model, trained on different data will thus expose different capabilities and can, therefore, solve different tasks. FL is a relatively recent distributed ML methodology aiming to bridge the gap between the need to train ever bigger ML models on ever larger datasets and the individual and companies’ will to protect and not share their private data. From another point of view, FL is also a way to distribute the training of an ML model even more than before. However, it should be considered that the learning performance of FL is usually lower than that of traditional centralised learning. This course will start from Kairouz ad McMahan’s definition of FL: ”Federated learning is a machine learning setting where multiple entities (clients) collaborate in solving a machine learning problem, under the coordination of a central server or service provider. Each client’s raw data is stored locally and not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective.” From this starting point, the most significant aspects of FL will be exposed and discussed. This tutorial will particularly explore FL from both the learning and computational [5] performance perspectives, investigating its pros and cons in a distributed ML setting. Since FL natively targets data privacy, some insights on how the FL process can be attacked and protected will also be discussed from a high-level perspective. Finally, a hands-on session will guide the participants in building a basic FL system, providing a better understanding of the major implementational difficulties of such a technique.
2023
Samuele Fonio
Benchmarking Federated Learning Frameworks for Medical Imaging Tasks Miscellaneous
Image Analysis and Processing - ICIAP 2023 - 22th International Conference - FedMed, 2023.
Abstract | Links | BibTeX | Tags: ai, eupilot, fl, icsc
@misc{23:iciap:benchmed,
title = {Benchmarking Federated Learning Frameworks for Medical Imaging Tasks},
author = {Samuele Fonio},
url = {https://datacloud.di.unito.it/index.php/s/sR7YeTGgfH4DtCR},
year = {2023},
date = {2023-09-01},
address = {Udine, Italy},
abstract = {This paper presents a comprehensive benchmarking study of various Federated Learning (FL) frameworks applied to the task of Medical Image Classification. The research specifically addresses the often neglected and complex aspects of scalability and usability in off-the-shelf FL frameworks. Through experimental validation using real case deployments, we provide empirical evidence of the performance and practical relevance of open source FL frameworks. Our findings contribute valuable insights for anyone interested in deploying a FL system, with a particular focus on the healthcare domain—an increasingly attractive field for FL applications.},
howpublished = {Image Analysis and Processing - ICIAP 2023 - 22th International Conference - FedMed},
keywords = {ai, eupilot, fl, icsc},
pubstate = {published},
tppubtype = {misc}
}
This paper presents a comprehensive benchmarking study of various Federated Learning (FL) frameworks applied to the task of Medical Image Classification. The research specifically addresses the often neglected and complex aspects of scalability and usability in off-the-shelf FL frameworks. Through experimental validation using real case deployments, we provide empirical evidence of the performance and practical relevance of open source FL frameworks. Our findings contribute valuable insights for anyone interested in deploying a FL system, with a particular focus on the healthcare domain—an increasingly attractive field for FL applications.
Gianluca Mittone, Samuele Fonio
Benchmarking Federated Learning Scalability Miscellaneous
2nd Italian Conference on Big Data and Data Science (ITADATA 2023), 2023.
Abstract | Links | BibTeX | Tags: ai, eupilot, fl, icsc
@misc{23:itadata:fl_scaling,
title = {Benchmarking Federated Learning Scalability},
author = {Gianluca Mittone and Samuele Fonio},
url = {https://datacloud.di.unito.it/index.php/s/QZGxC4X3s5LG5oT},
year = {2023},
date = {2023-09-01},
address = {Naples, Italy},
abstract = {Federated Learning (FL) is a widespread Machine Learning paradigm handling distributed Big Data. In this work, we demonstrate that different FL frameworks expose different scaling performances despite adopting the same technologies, highlighting the need for a more comprehensive study on the topic.},
howpublished = {2nd Italian Conference on Big Data and Data Science (ITADATA 2023)},
keywords = {ai, eupilot, fl, icsc},
pubstate = {published},
tppubtype = {misc}
}
Federated Learning (FL) is a widespread Machine Learning paradigm handling distributed Big Data. In this work, we demonstrate that different FL frameworks expose different scaling performances despite adopting the same technologies, highlighting the need for a more comprehensive study on the topic.
Bruno Casella, Samuele Fonio
Architecture-Based FedAvg for Vertical Federated Learning Miscellaneous
2023.
Abstract | Links | BibTeX | Tags: ai, epi, fl, icsc
@misc{23:casella:architecturalfedavgtalk,
title = {Architecture-Based FedAvg for Vertical Federated Learning},
author = {Bruno Casella and Samuele Fonio},
url = {https://datacloud.di.unito.it/index.php/s/kJQxnqG4d2ZSicK},
year = {2023},
date = {2023-01-01},
booktitle = {Proceedings of the 3rd Workshop on Distributed Machine Learning for the Intelligent Computing Continuum (DML-ICC), IEEE/ACM UCC 2023, Taormina, Italy, 4 December 2023},
abstract = {Federated Learning (FL) has emerged as a promising solution to address privacy concerns by collaboratively training Deep Learning (DL) models across distributed parties. This work proposes an architecture-based aggregation strategy in Vertical FL, where parties hold data with different attributes but shared instances. Our approach leverages the identical architectural parts, i.e. neural network layers, of different models to selectively aggregate weights, which is particularly relevant when collaborating with institutions holding different types of datasets, i.e., image, text, or tabular datasets. In a scenario where two entities train DL models, such as a Convolutional Neural Network (CNN) and a Multi-Layer Perceptron (MLP), our strategy computes the average only for architecturally identical segments. This preserves data-specific features learned from demographic and clinical data. We tested our approach on two clinical datasets, i.e., the COVID-CXR dataset and the ADNI study. Results show that our method achieves comparable results with the centralized scenario, in which all the data are collected in a single data lake, and benefits from FL generalizability. In particular, compared to the non-federated models, our proposed proof-of-concept model exhibits a slight performance loss on the COVID-CXR dataset (less than 8%), but outperforms ADNI models by up to 12%. Moreover, communication costs between training rounds are minimized by exchanging only the dense layer parameters.},
keywords = {ai, epi, fl, icsc},
pubstate = {published},
tppubtype = {misc}
}
Federated Learning (FL) has emerged as a promising solution to address privacy concerns by collaboratively training Deep Learning (DL) models across distributed parties. This work proposes an architecture-based aggregation strategy in Vertical FL, where parties hold data with different attributes but shared instances. Our approach leverages the identical architectural parts, i.e. neural network layers, of different models to selectively aggregate weights, which is particularly relevant when collaborating with institutions holding different types of datasets, i.e., image, text, or tabular datasets. In a scenario where two entities train DL models, such as a Convolutional Neural Network (CNN) and a Multi-Layer Perceptron (MLP), our strategy computes the average only for architecturally identical segments. This preserves data-specific features learned from demographic and clinical data. We tested our approach on two clinical datasets, i.e., the COVID-CXR dataset and the ADNI study. Results show that our method achieves comparable results with the centralized scenario, in which all the data are collected in a single data lake, and benefits from FL generalizability. In particular, compared to the non-federated models, our proposed proof-of-concept model exhibits a slight performance loss on the COVID-CXR dataset (less than 8%), but outperforms ADNI models by up to 12%. Moreover, communication costs between training rounds are minimized by exchanging only the dense layer parameters.