it is safe to recover from this. The fairseq-py guys have started doing this (in case there is a really long sequence length that comes in…).
Catch the python exception and then call torch.cuda.empty_cache() and continue business as usual.