I run into the same error when using a model from Huggingface transformers (BertModel). The code runs fine on CPU:
/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexT
ype>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, Ds
tDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [145,0,0], thread: [95,0,0] Assertion srcIndex < srcS electDimSize failed.
Traceback (most recent call last):
File “run.py”, line 58, in 
cmd(args)
File “/project/piqasso/tools/biaffine-parser/parser/cmds/train.py”, line 82, in call
self.train(train.loader)
File “/project/piqasso/tools/biaffine-parser/parser/cmds/cmd.py”, line 83, in train
arc_scores, rel_scores = self.model(words, feats)
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 532, in call
result = self.forward(*input, **kwargs)
File “/project/piqasso/tools/biaffine-parser/parser/model.py”, line 90, in forward
feat_embed = self.feat_embed(*feats)
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 532, in call
result = self.forward(*input, **kwargs)
File “/project/piqasso/tools/biaffine-parser/parser/modules/bert.py”, line 43, in forward
bert = bert[bert_mask].split(bert_lens[mask].tolist())
RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered
How can I figure out what is the culprit?