Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
5 views

2c PyTorch4

Uploaded by

jiejialing08
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
5 views

2c PyTorch4

Uploaded by

jiejialing08
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Typical Structure of a PyTorch Program

COMP9444: Neural Networks and # create neural network


net = MyNetwork().to(device) # CPU or GPU
Deep Learning
# prepare to load the training and test data
Week 2c. PyTorch train loader = torch.utils.data.DataLoader(...)
test loader = torch.utils.data.DataLoader(...)
Alan Blair
# choose between SGD, Adam or other optimizer
School of Computer Science and Engineering
optimizer = torch.optim.SGD(net.parameters,...)
June 3, 2024
for epoch in range(1, epochs): # training loop
train(args, net, device, train loader, optimizer)
# periodically evaluate network on test data
if epoch % 10 == 0:
test( args, net, device, test loader)

Defining a Network Structure Defining a Custom Model


This module computes a function of the form (x, y) !→ Ax log(y) + By 2
import torch.nn as nn
class MyNetwork(torch.nn.Module):
class MyModel(nn.Module):
def init (self): def init (self):
super(MyNetwork, self). init () super(MyModel, self). init ()
# define structure of the network here self.A = nn.Parameter(torch.randn((1),requires grad=True))
self.B = nn.Parameter(torch.randn((1),requires grad=True))
def forward(self, input):
# apply network and return output def forward(self, input):
output = self.A * input[:,0] * torch.log(input[:,1]) \
+ self.B * input[:,1] * input[:,1]
return output

3 4
Building a Net from Individual Components Defining a Sequential Network
class MyModel(torch.nn.Module):
class MyModel(torch.nn.Module):
def init (self, num input, num hid, num out):
def init (self):
super(MyModel, self). init ()
super(MyModel, self). init ()
self.main = nn.Sequential(
self.in to hid = torch.nn.Linear(2,2)
nn.Linear(num input, num hid),
self.hid to out = torch.nn.Linear(2,1)
nn.Tanh(),
def forward(self, input): nn.Linear(num hid, num out),
hid sum = self.in to hid(input) nn.Sigmoid()
hidden = torch.tanh(hid sum) )
out sum = self.hid to out(hidden) def forward(self, input):
output = torch.sigmoid(out sum) output = self.main(input)
return output return output

5 6

Sequential Components Declaring Data Explicitly

Network layers:
➛ nn.Linear()
➛ nn.Conv2d() (Week 4) import torch.utils.data
Intermediate Operators:
# input and target values for the XOR task
➛ nn.Dropout()
input = torch.Tensor([[0,0],[0,1],[1,0],[1,1]])
➛ nn.BatchNorm() (Week 4)
target = torch.Tensor([[0],[1],[1],[0]])
Activation Functions:
➛ nn.Sigmoid() xdata = torch.utils.data.TensorDataset(input,target)
➛ nn.Tanh() train loader = torch.utils.data.DataLoader(xdata,batch size=4)
➛ nn.ReLU() (Week 3)

7 8
Loading Data from a .csv File Custom Datasets

import pandas as pd from data import ImageFolder


df = pd.read csv("sonar.all-data.csv") # load images from a specified directory
df = df.replace(’R’,0) dataset = ImageFolder(folder, transform)
df = df.replace(’M’,1)
import torchvision.datasets as dsets
data = torch.tensor(df.values,dtype=torch.float32)
num input = data.shape[1] - 1 # download popular image datasets remotely
input = data[:,0:num input] mnistset = dsets.MNIST(...)
target = data[:,num input:num input+1] cifarset = dsets.CIFAR10(...)
dataset = torch.utils.data.TensorDataset(input,target) celebset = dsets.CelebA(...)

9 10

Choosing an Optimizer Training

# SGD stands for "Stochastic Gradient Descent" def train(args, net, device, train loader, optimizer):
optimizer = torch.optim.SGD( net.parameters(),
lr=0.01, momentum=0.9, for batch idx, (data,target) in enumerate(train loader):
weight decay=0.0001)
optimizer.zero grad() # zero the gradients
# Adam = Adaptive Moment Estimation (good for deep networks) output = net(data) # apply network
optimizer = torch.optim.Adam(net.parameters(),eps=0.000001, loss = ... # compute loss function
lr=0.01, betas=(0.5,0.999), loss.backward() # compute gradients
weight decay=0.0001) optimizer.step() # update weights

11 12
Loss Functions Testing

def test(args, net, device, test loader):


loss = torch.sum((output-target)*(output-target))
loss = F.nll loss(output,target) # (Week 3) with torch.no grad(): # suppress updating of gradients
net.eval() # toggle batch norm, dropout
loss = F.binary cross entropy(output,target) # (Week 3) for data, target in test loader:
loss = F.softmax(output,dim=1) # (Week 3) output = model(data)
test loss = ...
loss = F.log softmax(output,dim=1) # (Week 3) print(test loss)
net.train() # toggle batch norm, dropout back again

13 14

Computational Graphs Controlling the Computational Graph

PyTorch automatically builds a computational graph, enabling it to backpropagate


derivatives. If we need to stop the gradients from being backpropagated through a certain
variable (or expression) A, we can exclude it from the computational graph by
Every parameter includes .data and .grad components, for example: using:
A.data
A.detach()
A.grad
By default, loss.backward() discards the computational graph after computing
optimizer.zero grad() sets all .grad components to zero.
the gradients.
loss.backward() updates the .grad component of all Parameters by
If needed, we can force it to keep the computational graph by calling it this way:
backpropagating gradients through the computational graph.
loss.backward(retain graph=True)
optimizer.step() updates the .data components.

15 16

You might also like