Your model’s performance might depend on the batch size e.g. if it’s using batchnorm layers.
Since you are using a smaller batch size on each rank now, the performance could change, and you could check SyncBatchNorm if that’s indeed the root cause.