You could initialize the LSTM states using the currently incoming input shape instead of the global batch size.
If that’s not possible for some reason, you would have to remove these samples using drop_last=True in the DataLoader.