How to Setup a Multi-GPU Linux Machine for Deep Learning in 2024 | by Nika | May, 2024 – Niraranra – Niraranra

DEEP LEARNING WITH MULTIPLE GPUS

Great-fast setup of CUDA and PyTorch in minutes!

As Deep Finding out fashions (significantly LLMs) maintain getting larger, the need for further GPU memory (VRAM) is ever-increasing for rising them and using them regionally. Setting up or buying a multi-GPU machine is solely the first part of the issue. Most libraries and functions solely use a single GPU by default. Thus, the machine moreover should have acceptable drivers along with libraries which will leverage the multi-GPU setup.

This story provides a info on organize a multi-GPU (Nvidia) Linux machine with important libraries. This may occasionally hopefully stop some time on experimentation and get you started in your enchancment.

On the end, hyperlinks are equipped to well-liked open-source libraries which will leverage the multi-GPU setup for Deep Finding out.

Objective

Prepare a Multi-GPU Linux system with essential libraries equal to CUDA Toolkit and PyTorch to get started with Deep Finding out 🤖. The an identical steps moreover apply to a single GPU machine.

We’re going to arrange 1) CUDA Toolkit, 2) PyTorch and three) Miniconda to get started with Deep Finding out using frameworks equal to exllamaV2 and torchtune.

©️ All the libraries and information talked about on this story are open-source and/or publicly obtainable.

Getting Started

Look at the number of GPUs put in inside the machine using the nvidia-smi command inside the terminal. It should print an inventory of all the put in GPUs. If there’s a discrepancy or if the command doesn’t work, first arrange the Nvidia drivers to your mannequin of Linux. Make sure the nvidia-smi command prints an inventory of all the GPUs put in in your machine as confirmed above.

Adjust to this internet web page to place in Nvidia Drivers if not carried out already:

How to install the NVIDIA drivers on Ubuntu 22.04 — Linux Tutorials — Learn Linux Configuration– (Provide: linuxconfig.org)

Step-1 Arrange CUDA-Toolkit

💡 Look at for any current CUDA folder at usr/native/cuda-xx. That means a mannequin of CUDA is already put in. If you already have the required CUDA toolkit put in (take a look at with the nvcc command in your terminal) please skip to Step-2.

Look at the CUDA mannequin wished to your required PyTorch library: Start Locally | PyTorch (We’re placing in Arrange CUDA 12.1)

Go to CUDA Toolkit 12.1 Downloads | NVIDIA Developer to accumulate Linux directions to place in CUDA 12.1 (choose your OS mannequin and the corresponding “deb (native)” installer form).

The terminal directions for the underside installer will appear consistent with your chosen selections. Copy-paste and run them in your Linux terminal to place within the CUDA toolkit. As an example, for x86_64 Ubuntu 22, run the following directions by opening the terminal inside the downloads folder:

wget https://developer.acquire.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /and so forth/apt/preferences.d/cuda-repository-pin-600
wget https://developer.acquire.nvidia.com/compute/cuda/12.1.0/local_installers/cuda-repo-ubuntu2204-12-1-local_12.1.0-530.30.02-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-1-local_12.1.0-530.30.02-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get change
sudo apt-get -y arrange cuda

⚠️Whereas placing within the CUDA toolkit, the installer would possibly quick a kernel change. If any pop-up appears inside the terminal to exchange the kernel, press the esc button to cancel it. Don’t change the kernel all through this stage!— it may break your Nvidia drivers ☠️.

Restart the Linux machine after the arrange. The nvcc command will nonetheless not work. It’s important so as to add the CUDA set as much as PATH. Open the .bashrc file using the nano editor.

nano /home/$USER/.bashrc

Scroll to the underside of the .bashrc file and add these two traces:

 export PATH="/usr/native/cuda-12.1/bin:$PATH"
export LD_LIBRARY_PATH="/usr/native/cuda-12.1/lib64:$LD_LIBRARY_PATH"

💡 Phrase you may change cuda-12.1 to your put in CUDA mannequin, cuda-xx if wished ultimately , ‘xx’ being your CUDA mannequin.

Save the changes and shut the nano editor:

 To avoid wasting a lot of changes - On you keyboard, press the following: 
ctrl + o             --> save 
enter or return key  --> accept changes
ctrl + x             --> shut editor

Shut and reopen the terminal. Now the nvcc--version command should print the put in CUDA mannequin in your terminal.

Step-2 Arrange Miniconda

Sooner than we arrange PyTorch, it’s greater to place in Miniconda after which arrange PyTorch inside a Conda environment. Additionally it is useful to create a model new Conda environment for each enterprise.

Open the terminal inside the Downloads folder and run the following directions:

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
# provoke conda
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh

Shut and re-open the terminal. Now the conda command should work.

Step-3 Arrange PyTorch

(Non-compulsory) — Create a model new conda environment to your enterprise. You’ll be capable to substitute <environment-name> with the determine of your various. I usually determine it after my enterprise determine. 💡 You’ll want to use the conda activate <environment-name> and conda deactivate <environment-name> directions sooner than and after working in your enterprise.

conda create -n <environment-name> python=3.11
# activate the environment
conda activate <environment-name>

Arrange the PyTorch library to your CUDA mannequin. The subsequent directions are for cuda-12.1 which we put in:

pip3 arrange torch torchvision torchaudio

The above command is obtained from PyTorch arrange info — Start Locally | PyTorch .

After PyTorch arrange, take a look at the number of GPUs seen to PyTorch inside the terminal.

python
>> import torch
>> print(torch.cuda.device_count())
8

This might print the number of GPUs put in inside the system (8 in my case), and should additionally match the number of listed GPUs inside the nvidia-smi command.

Viola! you is perhaps all set to begin out working in your Deep Finding out initiatives that leverage a lot of GPUs 🥳.

1. 🤗 To get started, you’ll be capable to clone a most well-liked model from Hugging Face:

2. 💬 For inference (using LLM fashions), clone and arrange exllamav2 in a separate environment. This makes use of all your GPUs for sooner inference: (Look at my medium internet web page for an in depth tutorial)

3. 👨‍🏫 For fine-tuning or teaching, you’ll be capable to clone and arrange torchtune. Adjust to the instructions to each full finetune or lora finetune your fashions, leveraging all your GPUs: (Look at my medium internet web page for an in depth tutorial)

This info walks you through the machine setup wished for multi-GPU deep finding out. Now you may start engaged on any enterprise that leverages a lot of GPUs – like torchtune for sooner enchancment!

Preserve tuned for further detailed tutorials on exllamaV2 and torchtune.