Update: Sad…but I am pretty sure Out of Memory error is involved in this.
I have tried clearing cache, but no luck. (fairseq/trainer.py at 50a671f78d0c8de0392f924180db72ac9b41b801 · pytorch/fairseq · GitHub)
Currently, I have been using batch size 64, but I guess I might have to try batch size 32.
Still open to more solutions and tips.
P.s. My code is all vectorized.