Ha! zut, the problem still there the loss.backward() crash but I got a longer Traceback for my original script 
I use Pytorch '1.5.0.dev20200129. The crash do not occur neither on the shorter script I have given at the beginning of this thread nor on the smaller exercise given by @albanD.  Here is the Trace Back
Traceback (most recent call last):
  File "pz_train_kernelreg.py", line 562, in <module>
    main()
  File "pz_train_kernelreg.py", line 515, in main
    epoch, perturb=perturb)
  File "pz_train_kernelreg.py", line 133, in train
    loss.backward()
  File "...anaconda3/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/.../anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: tensor does not have a device (device at /opt/conda/conda-bld/pytorch_1580285301452/work/c10/core/TensorImpl.h:463)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x4e (0x2b0eb9c5c40e in /.../anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: at::Tensor::options() const + 0x1ff (0x2b0e81d805df in /.../anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #2: torch::autograd::generated::SliceBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0xea (0x2b0e8bb7e11a in .../anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x2e672fd (0x2b0e8c2e62fd in .../anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&) + 0x154b (0x2b0e8c2e277b in /.../anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&, bool) + 0x588 (0x2b0e8c2e3c88 in .../anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: torch::autograd::Engine::thread_init(int) + 0x39 (0x2b0e8c2dac99 in /.../anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #7: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x2b0e821d38a8 in .../anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0xc819d (0x2b0e7f93e19d in .../anaconda3/lib/python3.7/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #9: <unknown function> + 0x7e65 (0x2b0e60e87e65 in /lib64/libpthread.so.0)
frame #10: clone + 0x6d (0x2b0e6119a88d in /lib64/libc.so.6)