Did you make any progress with this?
I’m having a similar issue with torch.cholesky() inside MultivariateNormal.__init__()
I don’t believe I’m anywhere near running out of memory. Seems to happen somewhat randomly but always late in my training process.
torch.version.cuda returns '10.0.130' while nvidia-smi reports CUDA version 10.1 is this a problem?
So far unable to reproduce on CPU.
Below is what get’s reported using GPU (both with and without CUDA_LAUNCH_BLOCKING=1)
CUDA runtime error: an illegal memory access was encountered (77) in magma_spotrf_batched at /opt/conda/conda-bld/magma-cuda100_1564975479425/work/src/spotrf_batched.cpp:234
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda100_1564975479425/work/interface_cuda/interface.cpp:944
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda100_1564975479425/work/interface_cuda/interface.cpp:945
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda100_1564975479425/work/interface_cuda/interface.cpp:946
Traceback (most recent call last):
  File "/home/alex/repos/human-motion-rnn/lstm_motion_model/scripts/train", line 527, in <module>
      trainer.run(args)
  File "/home/alex/repos/human-motion-rnn/lstm_motion_model/scripts/train", line 513, in run
      validation_trajectories, save_directory, prefix=mode)
  File "/home/alex/repos/human-motion-rnn/lstm_motion_model/scripts/train", line 55, in train_and_validate
    model, optimizer, training_dataset, config)
   File "/home/alex/repos/human-motion-rnn/lstm_motion_model/scripts/train", line 192, in tbptt
      loss = model.compute_loss(predicted, target)
  File "/home/alex/repos/human-motion-rnn/lstm_motion_model/src/lstm_motion_model/models.py", line 1233, in compute_loss
      return super().compute_loss(predicted, target)
  File "/home/alex/repos/human-motion-rnn/lstm_motion_model/src/lstm_motion_model/models.py", line 281, in compute_loss
      return self.compute_loss_2d_gaussian(predicted, target)
  File "/home/alex/repos/human-motion-rnn/lstm_motion_model/src/lstm_motion_model/models.py", line 323, in compute_loss_2d_gaussian
      dist = MultivariateNormal(mu, covariance_matrix=covar)
  File "/home/alex/python-env/pytorch-env/lib/python3.6/site-packages/torch/distributions/multivariate_normal.py", line 149, in __init__
      self._unbroadcasted_scale_tril = torch.cholesky(covariance_matrix)
  RuntimeError: CUDA error: an illegal memory access was encountered
                > /home/alex/python-env/pytorch-env/lib/python3.6/site-packages/torch/distributions/multivariate_normal.py(149)__init__()
                -> self._unbroadcasted_scale_tril = torch.cholesky(covariance_matrix)

I’m on pytorch version 1.2.0