NVIDIA Workshop – Data Parallelism: How to train deep learning models on multiple GPUs

Computer Science Department, University of Torino

Bruno Casella, research fellow at the Computer Science Department, and NVIDIA DLI Certified Instructor, presents “Data Parallelism: How to Train Deep Learning Models on Multiple GPUs”.

Overview

Some of the most common challenges faced when training modern deep learning models involve leveraging increasingly large datasets and increasingly complex models. As a result, a growing and significant amount of computational power is required to train models effectively and efficiently. Learning how to distribute data across multiple GPUs during the training phase opens the door to new and important deep learning applications.

Goals

From participating in this workshop:

You will understand how the Data Parallelism technique across multiple GPUs is applied during the training of deep learning models.
You will achieve maximum throughput during training, maximizing the utilization of multiple GPUs.
You will distribute training across multiple GPUs using PyTorch Distributed Data Parallel.
You will understand and apply specific algorithmic considerations for multi-GPU training.

Agenda

9.30-10.00 Introduction and motivation
10.00-11.00 Lab 1: Neural network training and stochastic gradient descent
11.00-11.15 Coffee break
11.15-11.45 Introduction to data parallelism using PyTorch Distributed Data Parallel
11.45-13.00 Lab 2: Scaling to multiple GPUs with DistributedDataParallel
13.00-14.30 Lunch
14.30-15.00 Challenges of Data Parallel using multiple GPUs
15.00-16.30 Lab 3: Large scale optimizations.
16.30-16.45: Closing remarks.

Slides are available at this link.

Setup

To get started, follow these steps:

Create an NVIDIA Developer account at: https://learn.nvidia.com/dli-event.
Visit https://learn.nvidia.com/dli-event and enter the event code provided by the instructor.
You are ready to get started.

For more info: https://learn.nvidia.com/courses/course-detail?course_id=course-v1:DLI+C-MG-01+V3

Participation is free. Registration is mandatory.