Tensors

Contents

Tensors#

Note

Tensors are very similar to Numpy’s ndarray, with a few killer features added. First, the tensor class supports automatic differentiation. Second, It leverages GPUs to accelerate numerical computation, whereas Numpy only runs on CPUs. These properties make neural networks both easy to code and fast to run.

Getting Started#

PyTorch provides a variety of functions for creating new tensors prepopulated with values.

import torch

# evenly spaced values
x = torch.arange(12)
x

tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])

# total number of elements in a tensor
x.numel()

# a tensor's shape is the length along each axis
x.shape

torch.Size([12])

# change the shape of a tensor without altering its size or values
X = x.reshape(3, 4)
X

tensor([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]])

# contain all 0s
torch.zeros(2, 3, 4)

tensor([[[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]],

        [[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]]])

# contain all 1s
torch.ones(2, 3, 4)

tensor([[[1., 1., 1., 1.],
         [1., 1., 1., 1.],
         [1., 1., 1., 1.]],

        [[1., 1., 1., 1.],
         [1., 1., 1., 1.],
         [1., 1., 1., 1.]]])

# standard Gaussian distribution
torch.randn(3, 4)

tensor([[ 1.5420,  1.6180, -0.2885, -0.1658],
        [-0.9950, -1.3968,  0.3832,  0.0555],
        [ 1.6536,  1.3877, -1.2579,  1.1887]])

# from Python lists
torch.tensor([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])

tensor([[2, 1, 4, 3],
        [1, 2, 3, 4],
        [4, 3, 2, 1]])

Indexing and Slicing#

# access by indexing
X[-1]

tensor([ 8,  9, 10, 11])

# access by slicing
X[1:3]

tensor([[ 4,  5,  6,  7],
        [ 8,  9, 10, 11]])

# write elements
X[1, 2] = 17
X

tensor([[ 0,  1,  2,  3],
        [ 4,  5, 17,  7],
        [ 8,  9, 10, 11]])

Operations#

x = torch.tensor([1.0, 2, 4, 8])
y = torch.tensor([2, 2, 2, 2])
# ** means exponentiation
x + y, x - y, x * y, x / y, x ** y

(tensor([ 3.,  4.,  6., 10.]),
 tensor([-1.,  0.,  2.,  6.]),
 tensor([ 2.,  4.,  8., 16.]),
 tensor([0.5000, 1.0000, 2.0000, 4.0000]),
 tensor([ 1.,  4., 16., 64.]))

torch.exp(x)

tensor([2.7183e+00, 7.3891e+00, 5.4598e+01, 2.9810e+03])

We can also concatenate multiple tensors, stacking them end-to-end to form a larger one. We just need to provide a list of tensors and tell the system along which axis to concatenate.

X = torch.arange(12, dtype=torch.float32).reshape(3, 4)
Y = torch.tensor([[2.0, 1, 4, 3], [4, 3, 2, 1], [1, 2, 3, 4]])
# along rows
torch.cat((X, Y), dim=0)

tensor([[ 0.,  1.,  2.,  3.],
        [ 4.,  5.,  6.,  7.],
        [ 8.,  9., 10., 11.],
        [ 2.,  1.,  4.,  3.],
        [ 4.,  3.,  2.,  1.],
        [ 1.,  2.,  3.,  4.]])

# along columns
torch.cat((X, Y), dim=1)

tensor([[ 0.,  1.,  2.,  3.,  2.,  1.,  4.,  3.],
        [ 4.,  5.,  6.,  7.,  4.,  3.,  2.,  1.],
        [ 8.,  9., 10., 11.,  1.,  2.,  3.,  4.]])

# construct a binary tensor via logical statements
X == Y

tensor([[False,  True, False,  True],
        [ True, False, False, False],
        [False, False, False, False]])

X.sum()

tensor(66.)

Broadcasting#

Two tensors are “broadcastable” if the following rules hold:

Each tensor has at least one dimension.
When iterating over the dimention sizes, starting at the trailing dimension, the dimension size must either be equal, one of them if 1, or one of them does not exists.

# x and y are not broadcastable, because x does not have at least 1 dimension.
x = torch.empty((0,))
y = torch.empty(2,2)

# x and y are broadcastable.
# 1st trailing dimension: x has size 1
# 2nd trailing dimension: y dimension does't exist
x = torch.empty(3,1)
y = torch.empty(  2)

# x and y are broadcastable.
# 1st trailing dimension: x has size 1
# 2nd trailing dimension: y has size 1
x = torch.empty(3,1)
y = torch.empty(1,2)

# x and y are not broadcastable, because in the 1st trailing dimension 2 != 3
x = torch.empty(1,2)
y = torch.empty(  3)

Saving memory#

# allocating new memory
before = id(X)
X = Y + X
id(x) == before

False

# use X[:] = <expression>
# inplace
before = id(Y)
X[:] = Y + X
id(Y) == before

True

# inplace
before = id(X)
X += Y
id(X) == before

True

Conversion to other Python Objects#

A = X.numpy()
B = torch.tensor(A)
type(A), type(B)

(numpy.ndarray, torch.Tensor)

# convert size-1 tensor to a Python scalar
a = torch.tensor([3.5])
a, a.item(), float(a), int(a)

(tensor([3.5000]), 3.5, 3.5, 3)