

# Linear algebra basics: vectors and matrices

A vector is a list of numbers, usually represented as a column or row. For example, a 2D vector can be represented as:

$$
\begin{bmatrix}
x \\
y
\end{bmatrix}
$$

A matrix is a 2D array of numbers, usually represented as a grid. For example, a 2x2 matrix can be represented as:

$$
\begin{bmatrix}
a & b \\
c & d
\end{bmatrix}
$$

## Exercise
Instructions:

Create a 3x3 matrix using the following numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9.

### Solution
$$
\begin{bmatrix}
1 & 2 & 3 \\
4 & 5 & 6 \\
7 & 8 & 9
\end{bmatrix}
$$



# Multidimensional arrays and tensor operations

A tensor can have any number of dimensions, but we'll focus on 2D, 3D, and 4D tensors in this section. Tensors are represented as arrays of arrays, with each dimension having a specific size.

For example, a 2x2x2 tensor can be represented as:

$$
\begin{bmatrix}
\begin{bmatrix}
a & b \\
c & d
\end{bmatrix} &
\begin{bmatrix}
e & f \\
g & h
\end{bmatrix} \\
\begin{bmatrix}
i & j \\
k & l
\end{bmatrix} &
\begin{bmatrix}
m & n \\
o & p
\end{bmatrix}
\end{bmatrix}
$$

## Exercise
Instructions:

Create a 3x3x3 tensor using the following numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9.

### Solution
$$
\begin{bmatrix}
\begin{bmatrix}
1 & 2 & 3 \\
4 & 5 & 6
\end{bmatrix} &
\begin{bmatrix}
7 & 8 & 9 \\
10 & 11 & 12
\end{bmatrix} \\
\begin{bmatrix}
13 & 14 & 15 \\
16 & 17 & 18
\end{bmatrix} &
\begin{bmatrix}
19 & 20 & 21 \\
22 & 23 & 24
\end{bmatrix}
\end{bmatrix}
$$



# Autograd and gradient computation

To use autograd in PyTorch, we need to enable the gradient computation by calling the `requires_grad` method on a tensor. For example:

```python
import torch

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
```

Once the gradient computation is enabled, we can perform various tensor operations and compute the gradients. For example, let's compute the gradient of the sum of the tensor with respect to its elements:

```python
y = x.sum()
y.backward()
```

The gradient of the sum with respect to each element of the tensor can be accessed using the `grad` attribute:

```python
print(x.grad)
```

## Exercise
Instructions:

1. Create a tensor of size 3x3 with elements from 1 to 9.
2. Enable gradient computation for the tensor.
3. Compute the sum of the tensor.
4. Compute the gradient of the sum with respect to the tensor's elements.
5. Print the gradient.

### Solution
```python
import torch

x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]], requires_grad=True)
y = x.sum()
y.backward()
print(x.grad)
```



# Optimization techniques for tensor computations

One common optimization technique is gradient descent. In PyTorch, we can use the `torch.optim` module to implement various optimization algorithms, such as stochastic gradient descent, Adam, and RMSprop.

For example, let's implement gradient descent to minimize the sum of a tensor:

```python
import torch
import torch.optim as optim

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
optimizer = optim.SGD([x], lr=0.1)

for epoch in range(1000):
    optimizer.zero_grad()
    y = x.sum()
    y.backward()
    optimizer.step()

print(x)
```

## Exercise
Instructions:

1. Create a tensor of size 3x3 with elements from 1 to 9.
2. Enable gradient computation for the tensor.
3. Implement gradient descent to minimize the sum of the tensor.
4. Print the optimized tensor.

### Solution
```python
import torch
import torch.optim as optim

x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]], requires_grad=True)
optimizer = optim.SGD([x], lr=0.1)

for epoch in range(1000):
    optimizer.zero_grad()
    y = x.sum()
    y.backward()
    optimizer.step()

print(x)
```



# Introduction to neural networks and their use with tensors

In PyTorch, we can create a neural network using the `torch.nn` module. For example, let's create a simple feedforward neural network with one hidden layer:

```python
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.hidden = nn.Linear(3, 2)
        self.output = nn.Linear(2, 1)

    def forward(self, x):
        x = torch.sigmoid(self.hidden(x))
        x = self.output(x)
        return x

net = Net()
```

## Exercise
Instructions:

1. Create a tensor of size 2x3 with random elements.
2. Pass the tensor through the neural network.
3. Print the output tensor.

### Solution
```python
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.hidden = nn.Linear(3, 2)
        self.output = nn.Linear(2, 1)

    def forward(self, x):
        x = torch.sigmoid(self.hidden(x))
        x = self.output(x)
        return x

net = Net()

x = torch.randn(2, 3)
y = net(x)
print(y)
```



# Creating custom layers and models in PyTorch

For example, let's create a custom layer that adds a constant value to the input tensor:

```python
import torch
import torch.nn as nn

class AddLayer(nn.Module):
    def __init__(self, constant):
        super(AddLayer, self).__init__()
        self.constant = constant

    def forward(self, x):
        return x + self.constant

add_layer = AddLayer(1.0)
```

## Exercise
Instructions:

1. Create a tensor of size 2x3 with random elements.
2. Pass the tensor through the custom layer.
3. Print the output tensor.

### Solution
```python
import torch
import torch.nn as nn

class AddLayer(nn.Module):
    def __init__(self, constant):
        super(AddLayer, self).__init__()
        self.constant = constant

    def forward(self, x):
        return x + self.constant

add_layer = AddLayer(1.0)

x = torch.randn(2, 3)
y = add_layer(x)
print(y)
```



# Training and evaluating neural networks with PyTorch

For example, let's train a simple feedforward neural network on the MNIST dataset:

```python
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.hidden = nn.Linear(784, 128)
        self.output = nn.Linear(128, 10)

    def forward(self, x):
        x = torch.relu(self.hidden(x))
        x = self.output(x)
        return x

net = Net()

# Load and preprocess the MNIST dataset
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
train_dataset = datasets.MNIST('data', train=True, download=True, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)

# Train the neural network
optimizer = optim.Adam(net.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

for epoch in range(10):
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        output = net(data.view(-1, 784))
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

# Evaluate the neural network
test_dataset = datasets.MNIST('data', train=False, download=True, transform=transform)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=True)
correct = 0
total = 0

with torch.no_grad():
    for data, target in test_loader:
        output = net(data.view(-1, 784))
        _, predicted = torch.max(output.data, 1)
        total += target.size(0)
        correct += (predicted == target).sum().item()

print('Accuracy: {:.2f}%'.format(100 * correct / total))
```

## Exercise
Instructions:

1. Train a simple feedforward neural network on the MNIST dataset.
2. Evaluate the neural network on the test dataset.
3. Print the accuracy of the neural network.

### Solution
```python
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.hidden = nn.Linear(784, 128)
        self.output = nn.Linear(128, 10)

    def forward(self, x):
        x = torch.relu(self.hidden(x))
        x = self.output(x)
        return x

net = Net()

# Load and preprocess the MNIST dataset
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
train_dataset = datasets.MNIST('data', train=True, download=True, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)

# Train the neural network
optimizer = optim.Adam(net.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

for epoch in range(10):
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        output = net(data.view(-1, 784))
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

# Evaluate the neural network
test_dataset = datasets.MNIST('data', train=False, download=True, transform=transform)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=True)
correct = 0
total = 0

with torch.no_grad():
    for data, target in test_loader:
        output = net(data.view(-1, 784))
        _, predicted = torch.max(output.data, 1)
        total += target.size(0)
        correct += (predicted == target).sum().item()

print('Accuracy: {:.2f}%'.format(100 * correct / total))
```



# Advanced tensor operations: broadcasting and slicing

Broadcasting is a technique that allows us to perform element-wise operations on tensors with different shapes. For example, let's perform element-wise addition between two tensors:

```python
import torch

a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])

c = a + b
```

Slicing is a technique that allows us to extract subsets of tensors. For example, let's extract the first row and second column of a tensor:

```python
import torch

x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

row = x[0]
column = x[:, 1]
```

## Exercise
Instructions:

1. Create two tensors of size 3x1 and 1x3, respectively.
2. Perform element-wise addition between the tensors.
3. Print the result.
4. Create a tensor of size 3x3.
5. Extract the first row and second column of the tensor.
6. Print the extracted rows and columns.

### Solution
```python
import torch

a = torch.tensor([1, 2, 3])
b = torch.tensor([[4], [5], [6]])

c = a + b
print(c)

x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

row = x[0]
column = x[:, 1]

print(row)
print(column)
```



# Using GPU acceleration with PyTorch

To use GPU acceleration in PyTorch, we need to move tensors and models to the GPU using the `to` method. For example, let's move a tensor and a neural network to the GPU:

```python
import torch

x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
net = Net().to('cuda')

x = x.to('cuda')
```

## Exercise
Instructions:

1. Create a tensor of size 2x3.
2. Move the tensor to the GPU.
3. Create a simple feedforward neural network.
4. Move the neural network to the GPU.

### Solution
```python
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.hidden = nn.Linear(3, 2)
        self.output = nn.Linear(2, 1)

    def forward(self, x):
        x = torch.sigmoid(self.hidden(x))
        x = self.output(x)
        return x

x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
net = Net().to('cuda')

x = x.to('cuda')
```



# Tensor operations for computer vision applications

For example, let's resize an image tensor to a smaller size:

```python
import torch
import torchvision.transforms as transforms

transform = transforms.Compose([transforms.Resize((32, 32))])

image = torch.randn(3, 100, 100)
resized_image = transform(image)
```

## Exercise
Instructions:

1. Load an image from a file.
2. Resize the image to a smaller size.
3. Print the resized image.

### Solution
```python
import torch
import torchvision.transforms as transforms
from PIL import Image

transform = transforms.Compose([transforms.Resize((32, 32))])

image = Image.open('path/to/image.jpg')
image = transforms.ToTensor()(image)

resized_image = transform(image)
print(resized_image)
```



# Tensor operations for natural language processing

For example, let's create a tensor of word embeddings using the GloVe pre-trained word embeddings:

```python
import torch
import torchtext.vocab as vocab

glove = torchtext.vocab.GloVe(name='6B', dim=300)

text = "This is an example sentence."
words = text.split()
word_indices = [glove.stoi[word] for word in words]

word_embeddings = torch.stack([glove.vectors[index] for index in word_indices])
```

## Exercise
Instructions:

1. Load a pre-trained word embedding model.
2. Tokenize a text sentence.
3. Convert the tokens to word indices.
4. Create a tensor of word embeddings for the sentence.
5. Print the word embeddings.

### Solution
```python
import torch
import torchtext.vocab as vocab

glove = torchtext.vocab.GloVe(name='6B', dim=300)

text = "This is an example sentence."
words = text.split()
word_indices = [glove.stoi[word] for word in words]

word_embeddings = torch.stack([glove.vectors[index] for index in word_indices])
print(word_embeddings)
```

