Thanks for the response, it also made me realize that it’s not entirely about stationarity, but rather about the fact that my production setting requires single sequential real-time predictions.
Even though this improved my results, I’ve now realized that the better option for my use case is group normalization (GroupNorm) and not using batch normalization at all. Gives better results and no gimmicks required.