It’s a custom RNN decoder for image captioning, described in this paper: https://arxiv.org/abs/1707.07998 16. Suppose I could give the full source code, though really wish I could provide a simple minimal working example.
Also if I do catch this exception and move onto the next iteration, I get this CUDA runtime error just trying to even move a tensor to the GPU for initialization.
File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 106, in init_state
state_attention = (hidden_attention_lstm.cuda(), cell_attention_lstm.cuda())
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/autograd/variable.py”, line 298, in cuda
return CudaTransfer.apply(self, device, async)
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py”, line 201, in forward
return i.cuda(async=async)
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/_utils.py”, line 69, in cuda
return new_type(self.size()).copy(self, async)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generic/THCTensorCopy.c:20
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [30,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.