Your cookie choices

We use necessary cookies to make this site work. With your permission, analytics cookies help us understand usage and marketing cookies support advertising.

You can reject non-essential cookies and still use the site. Change your choices anytime in Cookie settings.

Cookie policy
Cookie settings

Updated DeployCue Editorial Team

Setting up multi-GPU training

Learn how to split a training job across several GPUs, start a process for each GPU, check that the GPUs communicate, and recover from common errors.

Before you start

This guide assumes your training job already runs on a single GPU with a falling loss, the number that tracks how well the model is learning. Multi-GPU setups add new failure points, so fix single-GPU problems first. Check these items before you begin.

  • Every GPU should be the hardware model you planned for, as explained in GPU model names, and appear in the GPU status command.
  • The training framework, the GPU driver, and the collective communication library have matching versions on every machine.
  • Your script reads its rank, the number that identifies each process, and the world size, the total process count, from the launcher instead of hardcoding them.

Data parallel and model parallel

Data parallel training copies the full model onto every GPU. Each copy works on a different slice of the batch, the group of examples used for each update. After every step, the copies share their gradients, the values that say how to adjust the weights, so all copies stay identical.

Model parallel training splits the model itself across GPUs. Each device holds only some layers or weights, so a model too large for a single card can still run. The cost is that data must move between devices during every pass through the model. This makes the links between GPUs matter much more.

Most teams start with data parallel because it is simpler to set up and reason about. Move to model parallel only when the model no longer fits in memory on a single GPU.

Set up the run

The launcher starts a process for each GPU and gives every process its own rank. Work through the steps below in order. Each step confirms the step before it, so a problem shows up early instead of late in a long run.

  1. Launch the script through the launcher with a process for each GPU on the machine.
  2. Print the rank, the GPU name, and the world size from inside each process.
  3. Confirm that every process reports a different rank and a different GPU.
  4. Set the batch size for each GPU so the model and its working memory fit on the card.
  5. Run a few training steps and confirm that the loss falls.
  6. Start the full run only after the short test finishes without errors.

Check that the GPUs can communicate

Gradient sharing only works when GPUs can exchange data quickly across the interconnect, the physical link between devices. Most setups use a collective communication library, a set of routines that move and combine data across devices.

Run a small all-reduce test before the real job. An all-reduce adds the values from every GPU and sends the total back to all of them. When every process gets the expected total, the links work. If the test hangs, check the links and the GPU drivers first.

Watch the step time as well. If a step takes much longer on a multi-GPU run than on a single GPU, communication is the likely bottleneck.

When to move to several nodes

A node is a single machine with its own set of GPUs. Stay on a single node for as long as it can hold the work. Machines connected by a network synchronize more slowly than GPUs inside the same machine, so step time often grows when you add them.

Move to several nodes when the job needs more GPUs than a single machine offers, or when the run would take too long on what you have. Before you commit, ask the provider about the network type between machines, the speed of that network, and whether the nodes sit in the same facility.

What to do if it fails

Most failures appear in the first few steps, so read the first error message closely. The problems below are the ones teams meet most often.

  • Processes hang at startup. Check that every machine can reach the others on the ports the launcher uses and that host names resolve.
  • Out of memory on some GPUs only. Lower the batch size for each GPU and confirm the GPUs in use have the same memory size.
  • The loss turns into not-a-number values, called NaN. Lower the learning rate, the step size for each weight update, and check the input data for bad values.
  • A single GPU does all the work. Check that each process selects its own device, and that the script does not force every process onto the first device.
  • Throughput, meaning how much work finishes in a given time, is low. Measure communication time, check the link between GPUs, and confirm that data loading keeps up with training.

When the cause is unclear, run the same script on a single GPU. If it passes there, the problem sits in the multi-GPU setup rather than in the model code.

Where this matters when choosing a provider

Multi-GPU work depends on the hardware and network a provider offers, not only on the GPU model. Look at how GPUs connect inside a machine, how fast machines talk to each other, and whether nodes share a facility. The details above decide whether the setup in this guide runs well.

Compare what each provider offers, and ask for the network details in writing. Availability and terms vary by country and by provider, so check the provider's terms before moving data. Start with the GPU provider comparison, or read how to choose a GPU provider for the wider picture.