edshkim98:

If so, is there any alternative solution that I can do?


You can try LayerNorm as a substitute, but it is not always performing as well, being too different from batch norm. Or something like https://github.com/Cerebras/online-normalization 25