PyTorch Primer
This is post is inspired by Numpy Primer <http://suriyadeepan.github.io/2016-06-26-numpy-primer/>
Lets create some matrices
import torch x = torch.zeros(3, 3) print(x) 0 0 0 0 0 0 0 0 0 [torch.FloatTensor of size 3x3] x = torch.ones(3, 3) print(x) 1 1 1 1 1 1 1 1 1 [torch.FloatTensor of size 3x3] x = torch.randn(3, 3) print(x) -0.0008 -0.6619 -0.6790 0.9104 -0.1249 -0.4044 -1.0516 -0.4031 0.9166 [torch.FloatTensor of size 3x3]
I think now we can understand what the parameters to the above functions mean - the shape of the tensor. Take a look at non square matrices below
x = torch.zeros(2,4) print(x) 0 0 0 0 0 0 0 0 [torch.FloatTensor of size 2x4] x = torch.zeros(4,3) print(x) 0 0 0 0 0 0 0 0 0 0 0 0 [torch.FloatTensor of size 4x3]
How about a multidimensional vector - a tensor. Actually tensor is a general term for n-dimensional arrays like in numpy. If you were keen observant, you'd have notices by now that the output of every print(x) end with torch.FloatTensor. This term became famous with the deep learning storm.
x = torch.zeros(2, 3, 2, 2, 3) print(x) (0 ,0 ,0 ,.,.) = 0 0 0 0 0 0 (0 ,0 ,1 ,.,.) = 0 0 0 0 0 0 (0 ,1 ,0 ,.,.) = 0 0 0 0 0 0 (0 ,1 ,1 ,.,.) = 0 0 0 0 0 0 (0 ,2 ,0 ,.,.) = 0 0 0 0 0 0 (0 ,2 ,1 ,.,.) = 0 0 0 0 0 0 (1 ,0 ,0 ,.,.) = 0 0 0 0 0 0 (1 ,0 ,1 ,.,.) = 0 0 0 0 0 0 (1 ,1 ,0 ,.,.) = 0 0 0 0 0 0 (1 ,1 ,1 ,.,.) = 0 0 0 0 0 0 (1 ,2 ,0 ,.,.) = 0 0 0 0 0 0 (1 ,2 ,1 ,.,.) = 0 0 0 0 0 0 [torch.FloatTensor of size 2x3x2x2x3]
Pause for a moment and take a long look into how the tensor is printed. And then proceed to look for more matrices below. The identity matrix - matrix is the keyword.
# The identity matrix - max upto two dimensions x = torch.eye(3) print(x) 1 0 0 0 1 0 0 0 1 [torch.FloatTensor of size 3x3] x = torch.eye(3,4) print(x) 1 0 0 0 0 1 0 0 0 0 1 0 [torch.FloatTensor of size 3x4]
Alright I get it. Just ones and zeros are boring. Want some more numbers? torch.linspace(start, end, count) creates a list of numbers starting with start to end at the interval of (end - start)/(count - 1)
x = torch.linspace(1, 6, 10) print(x) 1.0000 1.5556 2.1111 2.6667 3.2222 3.7778 4.3333 4.8889 5.4444 6.0000 [torch.FloatTensor of size 10]
torch.logspace() is similar to torch.linspace() but in logarthmic steps.
x = torch.logspace(.1, 1, 5) print(x) 1.2589 2.1135 3.5481 5.9566 10.0000 [torch.FloatTensor of size 5]
Arithmetics
So what we created bunch of numbers. What is the use? Lets do some arithmetic
Elementwise addition
x = torch.ones(3) print(x) 1 1 1 [torch.FloatTensor of size 3] y = x + 2 print(y) 3 3 3 [torch.FloatTensor of size 3]
Elementwise multiplication
x = torch.range(1, 10) print(x) 1 2 3 4 5 6 7 8 9 10 [torch.FloatTensor of size 10] y = x * 2 print(y) 2 4 6 8 10 12 14 16 18 20 [torch.FloatTensor of size 10]
Iterating over the elements and multiplying them by some number
for i in range(10): y[i] = x[i] * i print(y) 0 2 6 12 20 30 42 56 72 90 [torch.FloatTensor of size 10]
So yes we can multiply the tensors by ordinary numbers, there is no need for 'i' to be a tensor, lets take a larger tensors, and assign its values by hand. So far we have been using the numbers generated by function like ones(), zeros() and rand(). In most cases we need our favoruite numbers. How to do this? torch.Tensor() to the rescue.
x = torch.Tensor([1,3,5,6,78,3,67]) print(x) 1 3 5 6 78 3 67 [torch.FloatTensor of size 7] x = torch.Tensor([range(0, 10, 2), range(10, 20, 2), range(20, 25)]) print(x) 0 2 4 6 8 10 12 14 16 18 20 21 22 23 24 [torch.FloatTensor of size 3x5]
As we can see, the torch.Tensor() takes any iterable or iterables of iterable and makes a tensor out of it
Broadcasting Adding a scalar to a tensor or multiplying a tensor by a scalar is essentially same as adding or multiplying the tensor by another tensor with shape same as the original tensor, with all its elements being the scalar. Too many words.
x = torch.linspace(1, 10, 10) print(x) 1 2 3 4 5 6 7 8 9 10 [torch.FloatTensor of size 10] w = torch.Tensor([2]) print(w) 2 [torch.FloatTensor of size 1] # broadcasting https://discuss.pytorch.org/t/broadcasting-or-alternative-solutions/120 y = w.expand_as(x) * x print(y) 2 4 6 8 10 12 14 16 18 20 [torch.FloatTensor of size 10] w = torch.Tensor([2, 3]) print(w) 2 3 [torch.FloatTensor of size 2] y = w.expand_as(x) * x #Dont' work Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/paarulakan/environments/python/pytorch-py35/lib/python3.5/site-packages/torch/tensor.py", line 274, in expand_as return self.expand(tensor.size()) File "/home/paarulakan/environments/python/pytorch-py35/lib/python3.5/site-packages/torch/tensor.py", line 261, in expand raise ValueError('incorrect size: only supporting singleton expansion (size=1)') ValueError: incorrect size: only supporting singleton expansion (size=1)
Note that the expand_as() works only for singletons
Reshaping Tensors
view() takes list of number and reshapes the tensor as per the arguments.
The constrain is for view(a, b, c,....n) a x b x c x ... x n should be equal to the size of the given tensor. If you give '-1' as the last argument to view() it will calculate the last dimension by itself.
x = x.view(2, 5) print(x) 1 2 3 4 5 6 7 8 9 10 [torch.FloatTensor of size 2x5] # making matrices by hand is hard right? x = torch.range(1, 16) x_ = x.view(2, 8) print(x_) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [torch.FloatTensor of size 2x8] x_ = x.view(2, 2, 4) print(x_) (0 ,.,.) = 1 2 3 4 5 6 7 8 (1 ,.,.) = 9 10 11 12 13 14 15 16 [torch.FloatTensor of size 2x2x4] x_ = x.view(2, 2, 2, -1) print(x_) (0 ,0 ,.,.) = 1 2 3 4 (0 ,1 ,.,.) = 5 6 7 8 (1 ,0 ,.,.) = 9 10 11 12 (1 ,1 ,.,.) = 13 14 15 16 [torch.FloatTensor of size 2x2x2x2] x[5] = -1 * x[5] print(x_) (0 ,0 ,.,.) = 1 2 3 4 (0 ,1 ,.,.) = 5 -6 7 8 (1 ,0 ,.,.) = 9 10 11 12 (1 ,1 ,.,.) = 13 14 15 16 [torch.FloatTensor of size 2x2x2x2]
Notice that change in value of an element in x reflects in x_. It is because view() does exactly what is says it will do. It creates a view of the tensor, the underlying storage is same for x and x_
Statistics
x = torch.range(1, 16) x_ = x.view(4,4) print(x_) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [torch.FloatTensor of size 4x4] print(x_.sum(dim = 0)) 28 32 36 40 [torch.FloatTensor of size 1x4] print(x_.sum(dim = 1)) 10 26 42 58 [torch.FloatTensor of size 4x1] x_ = x.view(2,2,4) print(x_) (0 ,.,.) = 1 2 3 4 5 6 7 8 (1 ,.,.) = 9 10 11 12 13 14 15 16 [torch.FloatTensor of size 2x2x4] print(x_.sum(dim = 0)) (0 ,.,.) = 10 12 14 16 18 20 22 24 [torch.FloatTensor of size 1x2x4] print(x_.sum(dim = 1)) (0 ,.,.) = 6 8 10 12 (1 ,.,.) = 22 24 26 28 [torch.FloatTensor of size 2x1x4] print(x_.sum(dim = 2)) (0 ,.,.) = 10 26 (1 ,.,.) = 42 58 [torch.FloatTensor of size 2x2x1]
Matrix Multiplication
mm() is name. See how it is different from normal elementwise multiplication, like we used to do in linear algebra class?
#lets do some matrix multiplications w = torch.Tensor([[10, 20], [30, 40]]) x = torch.Tensor([[1,2], [3,4]]) print(w.mm(x)) 70 100 150 220 [torch.FloatTensor of size 2x2] w = torch.Tensor([[10, 20], [30, 40], [50, 60]]) x = torch.Tensor([[1,2,5], [3,4,6]]) print(w.mm(x)) 70 100 170 150 220 390 230 340 610 [torch.FloatTensor of size 3x3]
Indexing and slicing
#Indexing and slicing x = torch.range(1, 64) print(x) 1 2 3 4 5 . . . 61 62 63 64 [torch.FloatTensor of size 64] x_ = x.view(4,4,4) print(x_) (0 ,.,.) = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 (1 ,.,.) = 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 (2 ,.,.) = 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 (3 ,.,.) = 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 [torch.FloatTensor of size 4x4x4] print(x_[1, 1, :]) 21 22 23 24 [torch.FloatTensor of size 4]
Masking
#the expression 'x_ % 4 == 0' creates a mask print(x_ % 4 == 0) (0 ,.,.) = 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 (1 ,.,.) = 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 (2 ,.,.) = 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 (3 ,.,.) = 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 1 [torch.ByteTensor of size 4x4x4] # Use the mask to extract just those elements x_[0][(x_%4 == 0)[0]] 4 8 12 16 [torch.FloatTensor of size 4] #more grander masking example x = torch.range(1, 64) print(x) 1 2 3 4 5 . . . 61 62 63 64 [torch.FloatTensor of size 64] x_ = x.view(8,8) print(x_) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 [torch.FloatTensor of size 8x8] x_ind = torch.eye(8).byte() print(x_ind) 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 [torch.ByteTensor of size 8x8] print(x[x_ind]) 1 10 19 28 37 46 55 64 [torch.FloatTensor of size 8]
Please leave your comments below.
Comments