Hi,
I’m implementing an Unscented Kalman Filter in PyTorch 1.1.0  and I am having issues with the following function:
  def sigma_points(self, mu, sigma):

        U = torch.cholesky((self.l + self.n) * sigma)
        sigmas = [mu]

        for i in range(self.n):
            x1 = mu - U[:, :, i]
            x2 = mu + U[:, :, i]
            sigmas.extend([x1, x2])

        return torch.stack(sigmas, 1).view(-1, self.n)

Where sigma is a batched square matrix.
I get this error when running with CUDA_LAUNCH_BLOCKING=1
CUDA runtime error: an illegal memory access was encountered (77) in magma_spotrf_batched at /magma-2.5.0/src/spotrf_batched.cpp:234
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /magma-2.5.0/interface_cuda/interface.cpp:944
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /magma-2.5.0/interface_cuda/interface.cpp:945
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /magma-2.5.0/interface_cuda/interface.cpp:946
Traceback (most recent call last):
  File "run_pendulum_analogy.py", line 120, in <module>
    overshooting_d=args.overshooting_d, beta=args.beta, kl_anneal=args.kl_annealing)
  File "/home/tpower/dev/xdtl/pendulum_analogy/experiments/ukfvae.py", line 55, in run
    loss, recon, ql, ll, imgs = self.train(batch_size, train_obs, train_act)
  File "/home/tpower/dev/xdtl/pendulum_analogy/experiments/ukfvae.py", line 133, in train
    a, z_batch, prior_mu, prior_std)
  File "/home/tpower/dev/xdtl/pendulum_analogy/filters/ukf.py", line 80, in filter
    mu_bar, sigma_bar, measurement_fn)
  File "/home/tpower/dev/xdtl/pendulum_analogy/filters/ukf.py", line 23, in update
    sigma_points = self.sigma_point_selector.sigma_points(mu_bar, sigma_bar)
  File "/home/tpower/dev/xdtl/pendulum_analogy/filters/ukf.py", line 227, in sigma_points
    U = torch.cholesky((self.l + self.n) * sigma)
RuntimeError: CUDA error: an illegal memory access was encountered


I don’t see this error when running on CPU, and weirdly enough if I instead loop through and do torch.cholesky on each matrix in the batch individually, I do not get an error.
When I monitor nvidia-smi my memory usage is well below my maxmimum, so I don’t think I am running out of memory.
Thanks