If what you are doing here does not trigger any errors, it means that the GPU is working well, since all of your tensors are in the GPU.
So you know, what the code you posted is doing is calculating by ‘hand’ the error and then updating the values of w1 and w2.
Using the code you and ptrblck provided, here is a snippet you can copy and paste to check if the system is using the GPU for training:
import torch
import torch.optim as optim

model = torch.nn.Linear(5, 2)

#Initialize optimizer
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

device = 'cuda'
model.to(device)
data = torch.randn(1, 5, device=device)
out = model(data)
out.mean().backward()

optimizer.step()
optimizer.zero_grad()