Are you padding the input based on the largest sequence length?
If so, you might get unlucky and a batch might contain “short” examples, thus your activation might be too small at this particular layer.