Yes, that’s correct. Either you only run inference on the model on rank 0, or you explicitly replicate the BN stats from rank 0 to the other ranks prior to inference. The model weights should be identical across processes and only the BN stats should be different.