PyTorch CHEAT SHEET

1 Load data 2 Define model 3 Train model 4 Evaluate model

General Define model class Net(nn.Module):

def __init__():
Save/Load model
PyTorch is a open source machine learning framework. It uses torch.Tensor – multi-dimensional super(Net, self).__init__() model = torch.load('PATH') Load model
matrices – to process. A core feature of neural networks in PyTorch is the autograd package, There are several ways to
define a neural network in self.conv torch.save(model, 'PATH') Save model
which provides automatic derivative calculations for all operations on tensors. = nn.Conv2D( , , )
PyTorch, e.g. with
nn.Sequential (a), as a self.pool It is common practice to save only the model parameters, not the
import torch Root package torch.randn(*size) Create random tensor = nn.MaxPool2D( ) whole model using model.state_dict()
class (b) or using a
import torch.nn as nn Neural networks torch.Tensor(L) Create tensor from list combination of both. self.fc = nn.Linear( , )
from torchvision import Popular image datasets, tnsr.view(a,b, ...)Reshape tensor to
datasets, models, transforms architectures & transforms size (a, b, ...)
def forward(self, x):
import torch.nn.functional as F Collection of layers, requires_grad=True tracks computation history
activations & more for derivative calculations x = self.pool(
model = nn.Sequential( F.relu(self.conv(x))
nn.Conv2D( , , ) )
Layers nn.ReLU() GPU Training
nn.Linear(m, n): Fully Connected nn.ConvXd(m, n, s): X-dimensional x = x.view(-1, )
nn.MaxPool2D( ) device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
layer (or dense layer) from convolutional layer from m to n channels x = self.fc(x)
m to n neurons nn.Flatten()
with kernel size s; X {1, 2, 3}
nn.Linear( , ) return x If a GPU with CUDA support is available, computations are sent to
the GPU with ID 0 using model.to(device) or
nn.Flatten(): Flattens a contiguous nn.MaxPoolXd(s): X-dimensional pooling ) a model = Net() b inputs, labels = data[0].to(device), data[1].to(device).
range of dimensions into a tensor layer with kernel size s; X {1, 2, 3}

nn.Dropout(p=0.5): Randomly
Train model
nn.BatchNormXd(n): Normalizes a X-dimensional
sets input elements to zero during input batch with n features; X {1, 2, 3} LOSS FUNCTIONS
training to prevent overfitting
PyTorch already offers a bunch of different
loss fuctions, e.g.:
nn.Embedding(m, n): Lookup table nn.RNN/LSTM/GRU: Recurrent networks nn.L1Loss Mean absolute error
to map dictionary of size m to connect neurons of one layer with neurons of the
embedding vector of size n same or a previous layer nn.MSELoss Mean squared error (L2Loss)
nn.CrossEntropyLoss Cross entropy, e.g. for single-label
torch.nn offers a bunch of other building blocks. classification or unbalanced training set
A list of state-of-the-art architectures can be found at https://paperswithcode.com/sota.
nn.BCELoss Binary cross entropy, e.g. for multi-label
classification or autoencoders
Load data Activation functions OPTIMIZATION (torch.optim)
A dataset is represented by a class that Common activation functions include ReLU,
inherits from Dataset (resembles a list Optimization algorithms are used to update
Sigmoid and Tanh, but there are other activation weights and dynamically adapt the learning
of tuples of the form (features, label)). functions as well. rate with gradient descent, e.g.:
DataLoader allows to load a dataset optim.SGD
nn.ReLU() creates a nn.Module for example to be used in Stochastic gradient descent
without caring about its structure.
Sequential models. F.relu() ist just a call of the ReLU function optim.Adam Adaptive moment estimation
Usually the dataset is split into training e.g. to be used in the forward method.
optim.Adagrad Adaptive gradient
(e.g. 80%) and test data (e.g. 20%).
optim.RMSProp Root mean square prop
nn.ReLU() or F.relu()
Output between 0 and ∞,
most frequently used activation function
Evaluate model
The evaluation examines whether the model provides
nn.Sigmoid() or F.sigmoid() satisfactory results on previously withheld data.
Output between 0 and 1, Depending on the objective, different metrics are used,
often used for predicting probabilities such as acurracy, precision, recall, F1, or BLEU.

model.eval() Activates evaluation mode, some layers

nn.Tanh() or F.tanh() behave differently
Output between -1 and 1, torch.no_grad() Prevents tracking history, reduces memory
often used for classification with two classes usage, speeds up calculations

Stefan Seegerer, hi@stefanseegerer.de Matthias Zürl, matthias.zuerl@fau.de CC-BY-SA Last updated: 10/2021