Hi,
One possible way to deal with this might be to initialize the learning rate to be lower. This will prevent any outlier from having profound affects on the model weights.
I also suggest a learning rate scheduler like Cosine Annealing. This is because once the model starts to converge you don’t want an outlier to make changes to your model that will slow down convergence. Therefore, a learning rate scheduler can lower the learning rate gradually as epochs progress which could help.
Sarthak Jain