just want to update that it is not because of we have any GPU memory increase in LSTM layers, but the fact that it’s followed by FC layers where without sorting, the sequence length gets longer as input to FC, thus bigger FC and GPU RAM consumption.