Thanks tom for your reply. The learning rate was indeed set to 1.0 which is a bit high I guess but maybe not yet rediculous. Switching the learning rate back to 0.01 alone is not enough as it seems because I’m still getting my nans then.
While I have still trouble getting a proper standard deviation to work, scaling by only subtracting the mean from input data seems to mitigate the problem. So far the network loss is decreasing / the network is converging in the couple of tests I did with the current settings.
So the solution to the specific problem in my case seemed to be

Do not detach output
Scale input data

As for the standard deviation problem I will open another thread.
Thanks tom for your help!
Edit 1: Guess I was cheering too early once again… In my scaling method I am subtracting the input data’s mean from input and dividing by the input data’s standard deviation which happens to be broken. As soon as I remove the division by the broken std-Tensor I keep getting nans again 
Edit 2: Interestingly when printing the result of
torch::std(data, 0);
I get

[ Variable[CPUType]{} ]

Which I interpret as an empty Tensor with no values (?). Nevertheless division by that Tensor seems to scale values of the data Tensor which seems to fix the nan issue… weird. Is this correct expected behaviour?