ShilinHE:

Hi greaber, Thanks for your reply, I found that this problem happened when doing the evaluation(inference). Actually, during the inference, only one instance was used, which is pretty small. Also, I monitored the memory and it was not out-of-memory. I am really confused.


I had similar issue during evaluation only but as @greaber mentioned it was a memory issue. When I decrease the training batch size, it went away