Now I want to find the global minimun of the NN using GD with pytorch.
I believe what I’m trying to do must have been done a thousand times before, if not ten thousand times.

Have a look at this example here:
https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-tensors-and-autograd 22
It is updating the parameters manually in the step
with torch.no_grad():
    w1 -= learning_rate * w1.grad
    w2 -= learning_rate * w2.grad

to minimize the loss function, but you can also use the SGD optimizer, which is explained here: https://pytorch.org/docs/stable/optim.html#module-torch.optim 3
If you want to do GD specifically, not SGD, then each batch would basically be your entire training set (if you can fit it into memory). Otherwise, just accumulate the error before you do the manual update