Extending xFFL with Interoperable Model, Tensor, Sequence, and Expert Parallelism

Extend xFFL with additional parallelization dimensions and enable their interoperability within large-scale AI and federated learning workflows.

The candidate will: 1. Review parallelization techniques for large-scale AI models. 2. Analyze the current xFFL architecture and extension points. 3. Design abstractions for model, tensor, sequence, and expert parallelism. 4. Implement the selected parallelization mechanisms. 5. Enable interoperability and composition among the supported dimensions. 6. Evaluate correctness, scalability, communication overhead, and performance.

6 to 9 months according to the student's background.

• Advanced knowledge of Python
• Knowledge of distributed deep learning
• Familiarity with parallel programming and HPC

1. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, & Matei Zaharia (2021). Efficient large-scale language model training on GPU clusters using megatron-LM. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021 (pp. 58). ACM.