The batch size might be too small, if you are dealing with different distributions and the batchnorm layers might estimate noisy running stats.
Would it be possible to increase the batch size and lower the sequence length for the sake of debugging?
Alternatively you could also change the momentum to weight the updates a bit less.