You could wrap the forward and backward pass to free the memory if the current sequence was too long and you ran out of memory.
However, this code won’t magically work on all types of models, so if you encounter this issue on a model with a fixed size, you might just want to lower your batch size.