If this error seems to be raised “randomly”, this might point to e.g. a specifically large input batch.
If you are dealing with a variable sequence length, you might want to truncate the samples to a fixed size.
Also make sure you are not storing any tensors, which are still attached to the computation graph, during the training.