Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
269 views

Pytorch Tutorial PDF

The document introduces PyTorch, an open-source machine learning framework. It discusses three levels of abstraction in PyTorch: Tensors which are like NumPy arrays but can run on GPUs; Variables which are nodes in a computational graph that store data and gradients; and Modules which are neural network layers that may store weights. It also covers autograd for automatic differentiation, the nn module for defining neural network architectures, optimizers, extending PyTorch, popular datasets and transforms, and visualization tools.

Uploaded by

Pranit Shinde
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
269 views

Pytorch Tutorial PDF

The document introduces PyTorch, an open-source machine learning framework. It discusses three levels of abstraction in PyTorch: Tensors which are like NumPy arrays but can run on GPUs; Variables which are nodes in a computational graph that store data and gradients; and Modules which are neural network layers that may store weights. It also covers autograd for automatic differentiation, the nn module for defining neural network architectures, optimizers, extending PyTorch, popular datasets and transforms, and visualization tools.

Uploaded by

Pranit Shinde
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 27

Large-scale Intelligent Systems Laboratory

Large-scale Intelligent Systems Laboratory (Li Lab)


National Science Foundation Center for Big Learning (CBL)
Department of Electrical and Computer Engineering (ECE)
Department of Computer & Information Science & Engineering (CISE)

Pytorch Tutorial

Xiaoyong Yuan, Xiyao Ma


2018/01

some slides copied from cs231n.stanford.edu


Large-scale Intelligent Systems Laboratory

Introduction
■ Why we need deep learning frameworks
(1) Easily build big computational graphs
(2) Easily compute gradients in computational graphs
(3) Run it all efficiently on GPU
■ Why we need Pytorch?
(1) Easy to implement, code, and debug
(2) More flexible due to its dynamic computational graph.
(3) High execution efficiency, since it developed from C.
Large-scale Intelligent Systems Laboratory

PyTorch: Three Levels of Abstraction


Tensor: Like array in Numpy, but runs on GPU
Variable: Node in a computational graph; stores
data and gradient
Module: A neural network layer; may store state or learnable weights
Large-scale Intelligent Systems Laboratory

PyTorch: Tensors

■ PyTorch Tensors are just like numpy arrays,


but they can run on GPU.
■ No built-in notion of computational graph, or
gradients, or deep learning.
■ Here we fit a two-layer net using PyTorch
Tensors
Large-scale Intelligent Systems Laboratory

PyTorch: Tensors
To run on GPU, just cast tensors to a
cuda data type!
(E,g torch.cuda.FloatTensor)
Create random tensors for data and
weights.

Forward pass:
compute predictions and loss

Backward pass:
manually compute gradients

Gradient descent step on weights


Large-scale Intelligent Systems Laboratory

Tensor: Operations http://pytorch.org/docs/


master/index.html
Create a tensor:

Math Operation:

Other operations:
Large-scale Intelligent Systems Laboratory

PyTorch: Autograd
■ A PyTorch Variable is a node in a
computational graph
■ x.data is a Tensor
■ x.grad is a Variable of gradients
(same shape as x.data)
■ x.grad.data is a Tensor of gradients
■ PyTorch Tensors and Variables have
the same API!
■ Variables remember how they
were created (for backprop)
Large-scale Intelligent Systems Laboratory

PyTorch: Autograd
We will not want gradients (of loss)
with respect to data

Do want gradients with respect to


weights

Forward pass looks exactly the same


as the Tensor version, but everything
is a variable now

Compute gradient of loss with respect


to w1 and w2 (zero out grads first)

Make gradient step on weights


Large-scale Intelligent Systems Laboratory

PyTorch: New Autograd Functions


Large-scale Intelligent Systems Laboratory

PyTorch: New Autograd Functions

Define your own autograd functions


by writing forward and backward for
Tensors
Large-scale Intelligent Systems Laboratory

PyTorch: nn

Higher-level wrapper for working with neural


nets
Large-scale Intelligent Systems Laboratory

PyTorch: nn
Define our model as a sequence of
layers

nn also defines loss functions

Forward pass: feed data to model,


and prediction to loss function

Backward pass:
compute all gradients

Make gradient step on each model


parameter
Large-scale Intelligent Systems Laboratory

Example: Define your own Module

You can find more details about API in the following link:
http://pytorch.org/docs/master/index.html
Large-scale Intelligent Systems Laboratory

PyTorch: optim

Use an optimizer for different update


rules

Update all parameters after


computing gradients
Large-scale Intelligent Systems Laboratory

PyTorch: nn Define new Modules


Define our whole model as a single
Module

Initializer sets up two children


(Modules can contain modules)
Note: No need to define backward -
autograd will handle it

Define forward pass using child


modules and autograd ops on
Variables

Construct and train an instance of our


model
Large-scale Intelligent Systems Laboratory

MNIST Example
https://github.com/pytorch/examples/blob/master/mnist/main.py
Large-scale Intelligent Systems Laboratory

Extending PyTorch
■ torch.autograd
■ customize forward() and backward()
■ check gradient
■ from torch.autograd import
gradcheck
Large-scale Intelligent Systems Laboratory

Extending PyTorch
■ torch.autograd
■ Adding function to a
Module
Large-scale Intelligent Systems Laboratory

Extending PyTorch
■ C-extension
a. prepare your C code

header with all functions

c file
Large-scale Intelligent Systems Laboratory

Extending PyTorch
■ C-extension
b. build custom extension c. include it in your Python code

run it and get a new folder with a .so file


Large-scale Intelligent Systems Laboratory

Torchvision
■ popular datasets
■ cifar10, coco, lsun, mnist, ...
■ popular model architectures (pretrained)
■ alexnet, densenet, inception, resnet, squeezenet, vgg
■ common image transformations
■ Normalize, Scale, CenterCrop, Pad, RandomCrop,
RandomFlip, …
■ Compose
Large-scale Intelligent Systems Laboratory

Visualization
■ TensorboardX
■ pip install tensorboardX
■ tensorboard for pytorch (and chainer,
mxnet, numpy, ...)
■ Support scalar, image, histogram, audio,
text, graph, onnx_graph, embedding and
pr_curve
■ demo http://35.197.26.245:6006/

class tensorboardX.SummaryWriter(log_dir=None, comment='')


Large-scale Intelligent Systems Laboratory

Visualization
■ TensorboardX
a. Run the demo script: python demo.py
b. Use TensorBoard with tensorboard --logdir runs (needs to install
TensorFlow)
Large-scale Intelligent Systems Laboratory

Visualization
■ Other choices

Visdom Seaborn
Large-scale Intelligent Systems Laboratory

Onnx
■ Open Neural Network Exchange (ONNX)
■ an open source format to move models between tools
■ defines an extensible computation graph model, built-in operators and
standard data types
■ Hardware Optimizations
■ Supported Tools
Frameworks

Converters Runtimes
Large-scale Intelligent Systems Laboratory

References
1. http://pytorch.org/
2. http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture
8.pdf
3. https://github.com/pytorch/pytorch
Large-scale Intelligent Systems Laboratory

Thank
You!

You might also like