# PyTorch Primer

This is post is inspired by Numpy Primer <http://suriyadeepan.github.io/2016-06-26-numpy-primer/>

## Lets create some matrices

import torch x = torch.zeros(3, 3) print(x) 0 0 0 0 0 0 0 0 0 [torch.FloatTensor of size 3x3] x = torch.ones(3, 3) print(x) 1 1 1 1 1 1 1 1 1 [torch.FloatTensor of size 3x3] x = torch.randn(3, 3) print(x) -0.0008 -0.6619 -0.6790 0.9104 -0.1249 -0.4044 -1.0516 -0.4031 0.9166 [torch.FloatTensor of size 3x3]

I think now we can understand what the parameters to the above functions mean - the shape of the tensor. Take a look at non square matrices below

x = torch.zeros(2,4) print(x) 0 0 0 0 0 0 0 0 [torch.FloatTensor of size 2x4] x = torch.zeros(4,3) print(x) 0 0 0 0 0 0 0 0 0 0 0 0 [torch.FloatTensor of size 4x3]

How about a multidimensional vector - a tensor. Actually tensor is a general term for n-dimensional arrays like in numpy. If you were keen observant, you'd have notices by now that the output of every **print(x)** end with **torch.FloatTensor**. This term became famous with the deep learning storm.

x = torch.zeros(2, 3, 2, 2, 3) print(x) (0 ,0 ,0 ,.,.) = 0 0 0 0 0 0 (0 ,0 ,1 ,.,.) = 0 0 0 0 0 0 (0 ,1 ,0 ,.,.) = 0 0 0 0 0 0 (0 ,1 ,1 ,.,.) = 0 0 0 0 0 0 (0 ,2 ,0 ,.,.) = 0 0 0 0 0 0 (0 ,2 ,1 ,.,.) = 0 0 0 0 0 0 (1 ,0 ,0 ,.,.) = 0 0 0 0 0 0 (1 ,0 ,1 ,.,.) = 0 0 0 0 0 0 (1 ,1 ,0 ,.,.) = 0 0 0 0 0 0 (1 ,1 ,1 ,.,.) = 0 0 0 0 0 0 (1 ,2 ,0 ,.,.) = 0 0 0 0 0 0 (1 ,2 ,1 ,.,.) = 0 0 0 0 0 0 [torch.FloatTensor of size 2x3x2x2x3]

Pause for a moment and take a long look into how the tensor is printed. And then proceed to look for more matrices below. The identity matrix - matrix is the keyword.

# The identity matrix - max upto two dimensions x = torch.eye(3) print(x) 1 0 0 0 1 0 0 0 1 [torch.FloatTensor of size 3x3] x = torch.eye(3,4) print(x) 1 0 0 0 0 1 0 0 0 0 1 0 [torch.FloatTensor of size 3x4]

Alright I get it. Just ones and zeros are boring. Want some more numbers?
**torch.linspace(start, end, count)** creates a list of numbers starting with **start** to **end** at the interval of **(end - start)/(count - 1)**

x = torch.linspace(1, 6, 10) print(x) 1.0000 1.5556 2.1111 2.6667 3.2222 3.7778 4.3333 4.8889 5.4444 6.0000 [torch.FloatTensor of size 10]

**torch.logspace()** is similar to torch.linspace() but in logarthmic steps.

x = torch.logspace(.1, 1, 5) print(x) 1.2589 2.1135 3.5481 5.9566 10.0000 [torch.FloatTensor of size 5]

## Arithmetics

So what we created bunch of numbers. What is the use? Lets do some arithmetic

Elementwise addition

x = torch.ones(3) print(x) 1 1 1 [torch.FloatTensor of size 3] y = x + 2 print(y) 3 3 3 [torch.FloatTensor of size 3]

Elementwise multiplication

x = torch.range(1, 10) print(x) 1 2 3 4 5 6 7 8 9 10 [torch.FloatTensor of size 10] y = x * 2 print(y) 2 4 6 8 10 12 14 16 18 20 [torch.FloatTensor of size 10]

Iterating over the elements and multiplying them by some number

for i in range(10): y[i] = x[i] * i print(y) 0 2 6 12 20 30 42 56 72 90 [torch.FloatTensor of size 10]

So yes we can multiply the tensors by ordinary numbers, there is no need for 'i' to be a tensor, lets take a larger tensors, and assign its values by hand. So far we have been using the numbers generated by function like **ones(), zeros() and rand()**. In most cases we need our favoruite numbers. How to do this? **torch.Tensor()** to the rescue.

x = torch.Tensor([1,3,5,6,78,3,67]) print(x) 1 3 5 6 78 3 67 [torch.FloatTensor of size 7] x = torch.Tensor([range(0, 10, 2), range(10, 20, 2), range(20, 25)]) print(x) 0 2 4 6 8 10 12 14 16 18 20 21 22 23 24 [torch.FloatTensor of size 3x5]

As we can see, the torch.Tensor() takes any iterable or iterables of iterable and makes a tensor out of it

Broadcasting Adding a scalar to a tensor or multiplying a tensor by a scalar is essentially same as adding or multiplying the tensor by another tensor with shape same as the original tensor, with all its elements being the scalar. Too many words.

x = torch.linspace(1, 10, 10) print(x) 1 2 3 4 5 6 7 8 9 10 [torch.FloatTensor of size 10] w = torch.Tensor([2]) print(w) 2 [torch.FloatTensor of size 1] # broadcasting https://discuss.pytorch.org/t/broadcasting-or-alternative-solutions/120 y = w.expand_as(x) * x print(y) 2 4 6 8 10 12 14 16 18 20 [torch.FloatTensor of size 10] w = torch.Tensor([2, 3]) print(w) 2 3 [torch.FloatTensor of size 2] y = w.expand_as(x) * x #Dont' work Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/paarulakan/environments/python/pytorch-py35/lib/python3.5/site-packages/torch/tensor.py", line 274, in expand_as return self.expand(tensor.size()) File "/home/paarulakan/environments/python/pytorch-py35/lib/python3.5/site-packages/torch/tensor.py", line 261, in expand raise ValueError('incorrect size: only supporting singleton expansion (size=1)') ValueError: incorrect size: only supporting singleton expansion (size=1)

Note that the expand_as() works only for singletons

### Reshaping Tensors

**view()** takes list of number and reshapes the tensor as per the arguments.

The constrain is for view(a, b, c,....n) a x b x c x ... x n should be equal to the size of the given tensor. If you give *'-1'* as the last argument to **view()** it will calculate the last dimension by itself.

x = x.view(2, 5) print(x) 1 2 3 4 5 6 7 8 9 10 [torch.FloatTensor of size 2x5] # making matrices by hand is hard right? x = torch.range(1, 16) x_ = x.view(2, 8) print(x_) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [torch.FloatTensor of size 2x8] x_ = x.view(2, 2, 4) print(x_) (0 ,.,.) = 1 2 3 4 5 6 7 8 (1 ,.,.) = 9 10 11 12 13 14 15 16 [torch.FloatTensor of size 2x2x4] x_ = x.view(2, 2, 2, -1) print(x_) (0 ,0 ,.,.) = 1 2 3 4 (0 ,1 ,.,.) = 5 6 7 8 (1 ,0 ,.,.) = 9 10 11 12 (1 ,1 ,.,.) = 13 14 15 16 [torch.FloatTensor of size 2x2x2x2] x[5] = -1 * x[5] print(x_) (0 ,0 ,.,.) = 1 2 3 4 (0 ,1 ,.,.) = 5 -6 7 8 (1 ,0 ,.,.) = 9 10 11 12 (1 ,1 ,.,.) = 13 14 15 16 [torch.FloatTensor of size 2x2x2x2]

Notice that change in value of an element in **x** reflects in **x_**. It is because view() does exactly what is says it will do. It creates a view of the tensor, the underlying storage is same for **x** and **x_**

### Statistics

x = torch.range(1, 16) x_ = x.view(4,4) print(x_) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [torch.FloatTensor of size 4x4] print(x_.sum(dim = 0)) 28 32 36 40 [torch.FloatTensor of size 1x4] print(x_.sum(dim = 1)) 10 26 42 58 [torch.FloatTensor of size 4x1] x_ = x.view(2,2,4) print(x_) (0 ,.,.) = 1 2 3 4 5 6 7 8 (1 ,.,.) = 9 10 11 12 13 14 15 16 [torch.FloatTensor of size 2x2x4] print(x_.sum(dim = 0)) (0 ,.,.) = 10 12 14 16 18 20 22 24 [torch.FloatTensor of size 1x2x4] print(x_.sum(dim = 1)) (0 ,.,.) = 6 8 10 12 (1 ,.,.) = 22 24 26 28 [torch.FloatTensor of size 2x1x4] print(x_.sum(dim = 2)) (0 ,.,.) = 10 26 (1 ,.,.) = 42 58 [torch.FloatTensor of size 2x2x1]

### Matrix Multiplication

**mm()** is name. See how it is different from normal elementwise multiplication, like we used to do in linear algebra class?

#lets do some matrix multiplications w = torch.Tensor([[10, 20], [30, 40]]) x = torch.Tensor([[1,2], [3,4]]) print(w.mm(x)) 70 100 150 220 [torch.FloatTensor of size 2x2] w = torch.Tensor([[10, 20], [30, 40], [50, 60]]) x = torch.Tensor([[1,2,5], [3,4,6]]) print(w.mm(x)) 70 100 170 150 220 390 230 340 610 [torch.FloatTensor of size 3x3]

### Indexing and slicing

#Indexing and slicing x = torch.range(1, 64) print(x) 1 2 3 4 5 . . . 61 62 63 64 [torch.FloatTensor of size 64] x_ = x.view(4,4,4) print(x_) (0 ,.,.) = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 (1 ,.,.) = 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 (2 ,.,.) = 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 (3 ,.,.) = 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 [torch.FloatTensor of size 4x4x4] print(x_[1, 1, :]) 21 22 23 24 [torch.FloatTensor of size 4]

### Masking

#the expression 'x_ % 4 == 0' creates a mask print(x_ % 4 == 0) (0 ,.,.) = 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 (1 ,.,.) = 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 (2 ,.,.) = 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 (3 ,.,.) = 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 [torch.ByteTensor of size 4x4x4] # Use the mask to extract just those elements x_[0][(x_%4 == 0)[0]] 4 8 12 16 [torch.FloatTensor of size 4] #more grander masking example x = torch.range(1, 64) print(x) 1 2 3 4 5 . . . 61 62 63 64 [torch.FloatTensor of size 64] x_ = x.view(8,8) print(x_) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 [torch.FloatTensor of size 8x8] x_ind = torch.eye(8).byte() print(x_ind) 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 [torch.ByteTensor of size 8x8] print(x[x_ind]) 1 10 19 28 37 46 55 64 [torch.FloatTensor of size 8]

Please leave your comments below.

## Comments