I am not an expert on how pytorch works on GPU but my feeling is that it benefits the most when there are alot of convolutional layers with a lot of filters (this is because when you enable GPU there is the time to transfer stuff from CPU to GPU so if your computational gain is not too much it might be shadowed by the time loss for the transfer of information). Try changing, for instance, your network to one with larger convolutional layers.
Also try increasing your batch size, I have a feeling that pytorch will probably be able to harness the GPU more by doing large matrix multplications on it when the batch size is large. In google colaboratory you can experiment up to around 512 for the batchsize.
For instance:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(16, 32, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv3 = nn.Conv2d(32, 64, 5)
        self.fc1 = nn.Linear(64, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = (F.relu(self.conv3(x)))
        x = x.view(-1, 64)
        x = self.fc1(x)

Here there is less linear layers but more convolutional layers with larger filters.