Not sure why this happens, but the solution for me was to call .double() on the pytorch Tensors:
x_torch = torch.from_numpy(x).double()
y_torch = torch.from_numpy(y).double()

and also on the model, before calling train:
model.double()
model.train()