Distributed Model Training & Frameworks on GCP — Tensorflow and PyTorch w/ Deep Learning (LLMs) | by Jose M | Nov, 2023

As talked about, there are 2 types of Distributed Teaching: Model Parallelism and Data Parallelism. Every use parallel execution.

All kinds has many various algorithms, usually often known as strategies. There are 2 paradigms of talked about algorithms: synchronous teaching and asynchronous teaching.

Model Parallelism 🌌 & Data Parallelism 🪐

1. Data Parallelism: Model is replicated. Model is able to see further data at each teaching step. Data is sliced all through GPUs. Broadly adopted and most common.
2. Model Parallelism: Model is minimize up into many various fashions to hurry up teaching.
   – Related principally to very-large fashions. Occasion: Huge Language Fashions (LLMs)
   – Best for fashions with unbiased parts of computation that could be run in parallel. Principally utilized in evaluation.

Important concepts for every

• Cluster -> Nodes -> Teaching service -> Helpful useful resource allocation -> Group of replicas (worker pool) -> Duplicate
• Cluster & nodes: Carry-over concepts from kafka/kubernetes/and lots of others
• Duplicate: teaching job
• Worker pool: Group of replicas
• Worker devices: GPU, CPU, TPU

Reference in how model parameters are updated.

• Synchronous (all-reduce):
• Asynchronous (parameter server):

These are two operation (math) varieties which define the construction of sync vs. Async teaching strategies.

• All-reduce (sync): Operation which reduces a set of arrays on distributed staff to a single array which is then re-distro’d once more to each worker. Think about it as sequential computations all through the GPUs.
  – TensorFlow selects the precise All-reduce algorithm
• Parameter server (async):

Concepts

asd

Tensorflow Strategies

Synchronous teaching (usually often known as Mirrored)

• Mirrored Method: many accelerators on the related worker
  – tf.distribute.MirroredStrategy
• Multi-Worker Mirrored: many accelerators on plenty of staff
  – tf.distribute.experimental.MultiWorkerMirroredStrategy
• Central Storage: Solely utilized to environments with a single machine working plenty of GPUs.

Asynchronous teaching

Extreme Effectivity Computing (HPC) frameworks are typically used to ship Distributed Teaching.

Code: PyTorch 👨‍💻

Written using torch.nn.parallel.DistributedDataParallel to educate a Linear Model with the MNIST dataset.

import os
import torch
import torch.nn as nn
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torchvision import datasets, transforms
# Establishing the system and DDP
def setup(rank, world_size):
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'
# initialize the tactic group
dist.init_process_group("gloo", rank=rank, world_size=world_size)
def cleanup():
dist.destroy_process_group()
# DDP Teaching Carry out
def observe(rank, world_size, model, dataloader, criterion, optimizer):
setup(rank, world_size)
# DDP encapsulates the model
model = DDP(model.to(rank), device_ids=[rank])
for epoch in differ(10):  
for batch, (data, targets) in enumerate(dataloader):
data = data.to(rank)
targets = targets.to(rank)
# Forward Cross
outputs = model(data)
loss = criterion(outputs, targets)
# Backward Cross and Optimization
optimizer.zero_grad()
loss.backward()
optimizer.step()
print(f'Rank {rank}, Epoch {epoch}, Loss {loss.merchandise()}')
cleanup()
def important():
world_size = torch.cuda.device_count()
# Sample model and data loader, Change alongside together with your exact model and data loader
model = nn.Linear(10, 10)
dataloader = torch.utils.data.DataLoader(datasets.MNIST('../data', observe=True, receive=True, 
rework=transforms.Compose([transforms.ToTensor(), 
transforms.Normalize((0.1307,), (0.3081,))])),
batch_size=64, shuffle=True, num_workers=2)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
torch.multiprocessing.spawn(observe, args=(world_size, model, dataloader, criterion, optimizer), nprocs=world_size)
if __name__ == "__main__":
attempt:
important()
in addition to Exception as e:
print("An error occurred:", str(e))

Code: TensorFlow 👨‍💻

Using tf.distribute.MirroredStrategy to educate a Keras model using checkpoints, a TensorBoard callback, and monitoring validation loss.

import os
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import Dense
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, TensorBoard
from tensorflow.keras.datasets import mnist
def get_model():
"""
This function defines a straightforward sequential model
"""
model = keras.Sequential([
keras.layers.Flatten(input_shape=(28, 28)),
Dense(128, activation='relu'),
Dense(10, activation='softmax')
])
return model
def load_data():
"""
This function plenty the mnist data and preprocesses it
"""
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
return (x_train, y_train), (x_test, y_test)
def important():
# Define the approach
approach = tf.distribute.MirroredStrategy()
# Load and preprocess the data
(x_train, y_train), (x_test, y_test) = load_data()
# Checkpoint itemizing
checkpoint_dir="./training_checkpoints"
checkpoint_prefix = os.path.be a part of(checkpoint_dir, "ckpt_{epoch}")
# Open a way scope and description the model, compile and match inside this scope
with approach.scope():
model = get_model()
model.compile(optimizer="adam",
loss="sparse_categorical_crossentropy",
metrics=['accuracy'])
# Define callbacks for checkpointing, early stopping and TensorBoard
callbacks = [
ModelCheckpoint(filepath=checkpoint_prefix, save_weights_only=True),
EarlyStopping(monitor="val_loss", patience=3),
TensorBoard(log_dir="./logs")
]
model.match(x_train, y_train, epochs=10, callbacks=callbacks, validation_data=(x_test, y_test))
# Think about the model
loss, accuracy = model.think about(x_test, y_test)
print(f'Loss: {loss}, Accuracy: {accuracy}')
if __name__ == "__main__":
attempt:
important()
in addition to Exception as e:
print("An error occurred:", str(e))

tbd

“Parallelism” is an thought of Laptop computer Science. Further significantly distributed computing, and you’d moreover say a queueing methodology.

On this planet of Distributed Teaching, Parallel Execution is used.

Two helpful diagrams evaluating each execution varieties: