— 2025
Federated Learning for Foundation Models
Gianluca Mittone
Abstract
Federated Learning (FL) is a distributed machine learning paradigm that enables multiple clients to collaboratively train models while keeping raw data local. This lecture introduces the fundamental principles of FL and examines their application to foundation models, with particular emphasis on privacy, communication efficiency, data and computational heterogeneity, and system-level constraints. FL is presented as an approach for learning across decentralized data sources without directly transferring sensitive information, using focused model updates that are aggregated by a coordinating server. Privacy can be further strengthened through techniques such as differential privacy and secure aggregation. The lecture discusses the main FL deployment settings, distinguishing cross-device learning, involving potentially millions of intermittently available clients, from cross-silo learning, where organizations are generally more stable and highly available. These settings introduce challenges related to non-IID data, heterogeneous computational resources, limited connectivity, client availability, and communication overhead. Federated optimization therefore requires careful coordination between local computation, communication rounds, privacy requirements, and system constraints. Federated Averaging (FedAvg) is presented as a common optimization approach in which locally trained model parameters are periodically aggregated to produce a shared global model. A particular focus is placed on Cross-Facility Federated Learning (xFFL), which extends FL to geographically distributed HPC infrastructures. The approach uses hierarchical orchestration through StreamFlow, HPC-adapting middleware, and containerized PyTorch deployments to address differences in hardware, software environments, and infrastructure policies. Experiments with LLaMA-2 and LLaMA-3 demonstrate the application of federated learning to large language model workloads across HPC facilities. Overall, FL for foundation models is characterized as an interdisciplinary problem combining machine learning, optimization, privacy, security, cryptography, information theory, and distributed systems. The slides emphasize that effective FL systems require co-design of optimization algorithms with privacy, security, communication, computational, and deployment constraints, while careful evaluation is necessary to distinguish simulation results from behavior in real-world federated environments.
@misc{25:asdmsd:fl,
abstract = {Federated Learning (FL) is a distributed machine learning paradigm that enables multiple clients to collaboratively train models while keeping raw data local. This lecture introduces the fundamental principles of FL and examines their application to foundation models, with particular emphasis on privacy, communication efficiency, data and computational heterogeneity, and system-level constraints. FL is presented as an approach for learning across decentralized data sources without directly transferring sensitive information, using focused model updates that are aggregated by a coordinating server. Privacy can be further strengthened through techniques such as differential privacy and secure aggregation. The lecture discusses the main FL deployment settings, distinguishing cross-device learning, involving potentially millions of intermittently available clients, from cross-silo learning, where organizations are generally more stable and highly available. These settings introduce challenges related to non-IID data, heterogeneous computational resources, limited connectivity, client availability, and communication overhead. Federated optimization therefore requires careful coordination between local computation, communication rounds, privacy requirements, and system constraints. Federated Averaging (FedAvg) is presented as a common optimization approach in which locally trained model parameters are periodically aggregated to produce a shared global model. A particular focus is placed on Cross-Facility Federated Learning (xFFL), which extends FL to geographically distributed HPC infrastructures. The approach uses hierarchical orchestration through StreamFlow, HPC-adapting middleware, and containerized PyTorch deployments to address differences in hardware, software environments, and infrastructure policies. Experiments with LLaMA-2 and LLaMA-3 demonstrate the application of federated learning to large language model workloads across HPC facilities. Overall, FL for foundation models is characterized as an interdisciplinary problem combining machine learning, optimization, privacy, security, cryptography, information theory, and distributed systems. The slides emphasize that effective FL systems require co-design of optimization algorithms with privacy, security, communication, computational, and deployment constraints, while careful evaluation is necessary to distinguish simulation results from behavior in real-world federated environments.},
address = {Trieste, Italy},
author = {Mittone, Gianluca},
howpublished = {ICTP Advanced School on Foundation Models for Scientific Discovery},
keywords = {icsc, dyman, fl},
month = jul,
title = {Federated Learning for Foundation Models},
url = {https://datacloud.di.unito.it/index.php/s/5gzSkSgN8nzKz9D},
year = 2025,
bdsk-url-1 = {https://datacloud.di.unito.it/index.php/s/5gzSkSgN8nzKz9D}}