I believe the problem was due to a sentence of length 383, which was split into 672 wordpieces.
I traced te error to this call, in
lib64/python3.6/site-packages/torch/nn/modules/sparse.py
111 ->	def forward(self, input):
112 return F.embedding(
113 input, self.weight, self.padding_idx, self.max_norm,
114 self.norm_type, self.scale_grad_by_freq, self.sparse)
where input = arange(672)
(Pdb)
–Return–
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=700 : an illegal memory access was encountered
So it likely hits the limit of 512 of BERT inputs and breaks the memory.
I wonder why they don’t check this limit and avoid memory corruption.