A multi-GPU environment is a computing system with more than one GPU device. Although multiple interconnected machines with just one GPU can be considered a multi-GPU environment, we usually employ this term to describe environments with two or more GPUs per machine.

To understand how this environment works under the hood, we need to learn about the connectivity of the devices and technologies that are adopted to provide efficient communication across multiple GPUs.

However, before we dive into these topics, we will answer a disquieting question that has probably come to your mind: will we have access to an expensive environment like that? Yes, we will. But first, let’s briefly discuss the increasing popularity of multi-GPU environments.

The popularity of multi-GPU environments

Going back 10 years ago, it was inconceivable to think of a machine with more than one GPU. Besides the high cost of this device, the applicability of...