error:
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:278
How to trigger it:
print (logits)
about this tensor:
print (logits.shape)
torch.Size([32, 80, 7])
type (logits)
torch.Tensor
Full stack trace:
RuntimeError                              Traceback (most recent call last)
 in ()
----> 1 print (logits)
7 frames
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in repr(self)
128         # characters to replace unicode characters with.
129         if sys.version_info > (3,):
–> 130             return torch._tensor_str._str(self)
131         else:
132             if hasattr(sys.stdout, ‘encoding’):
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _str(self)
309                 tensor_str = _tensor_str(self.to_dense(), indent)
310             else:
–> 311                 tensor_str = _tensor_str(self, indent)
312
313     if self.layout != torch.strided:
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _tensor_str(self, indent)
207     if self.dtype is torch.float16 or self.dtype is torch.bfloat16:
208         self = self.float()
–> 209     formatter = _Formatter(get_summarized_data(self) if summarize else self)
210     return _tensor_str_with_formatter(self, indent, formatter, summarize)
211
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in get_summarized_data(self)
240         end = ([self[i]
241                for i in range(len(self) - PRINT_OPTS.edgeitems, len(self))])
–> 242         return torch.stack([get_summarized_data(x) for x in (start + end)])
243     else:
244         return torch.stack([get_summarized_data(x) for x in self])
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in (.0)
240         end = ([self[i]
241                for i in range(len(self) - PRINT_OPTS.edgeitems, len(self))])
–> 242         return torch.stack([get_summarized_data(x) for x in (start + end)])
243     else:
244         return torch.stack([get_summarized_data(x) for x in self])
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in get_summarized_data(self)
240         end = ([self[i]
241                for i in range(len(self) - PRINT_OPTS.edgeitems, len(self))])
–> 242         return torch.stack([get_summarized_data(x) for x in (start + end)])
243     else:
244         return torch.stack([get_summarized_data(x) for x in self])
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in (.0)
240         end = ([self[i]
241                for i in range(len(self) - PRINT_OPTS.edgeitems, len(self))])
–> 242         return torch.stack([get_summarized_data(x) for x in (start + end)])
243     else:
244         return torch.stack([get_summarized_data(x) for x in self])
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in get_summarized_data(self)
233     if dim == 1:
234         if self.size(0) > 2 * PRINT_OPTS.edgeitems:
–> 235             return torch.cat((self[:PRINT_OPTS.edgeitems], self[-PRINT_OPTS.edgeitems:]))
236         else:
237             return self
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/THCGeneral.cpp:371
This is triggered in evaluation mode. I could provide the code