Same issue with the newer cuda/cudnn version.
After about 20min of training, I get this:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-33f16e9e6467> in <module>
      6 trainer = Trainer(splitter, LeggedTCN)
      7 print("training")
----> 8 trainer.train()
      9 #with sprees

~/pycharms_remote/valx_trader/rrl_hm/training.py in train(self)
    281                 if self.flippers['mixup']: X, Y = self.mixup(X, Y, self.flippers['mixup'])
    282 
--> 283                 loss = self.train_iteration(X, Y, sym_perm, epoch) # crm,
    284                 self.iter_losses[sub_epoch] = loss
    285                 if batch_index % print_every == 0:

~/pycharms_remote/valx_trader/rrl_hm/training.py in train_iteration(self, X, Y, sym_perm, epoch)
    311         self.optimizer.step()
    312 
--> 313         self.losses[epoch] += loss.item()
    314         # print('loss', loss)
    315         return loss

RuntimeError: CUDA error: an illegal memory access was encountered