That’s a big topic, there is “vertical” and “horizontal” (h[t-1] -> h[t]) normalization, and many approaches have been tried… But basically, simple vertical batchnorm is problematic, LayerNorm(num_features) works, but feature distortion it does may sometimes have adverse effects. You can check Layer Normalization 28 paper, as it addresses rnn normalization. Recurrent batch normalization 27 paper describes horizontal LSTM normalization.