Thank you so much @osalpekar for your replay and for sharing the valuable information.

Were the training jobs with and without Apex done on the same machines? - No, I have used cloud instances to run the experiment with and without Apex

I’ll be redirecting this issue to Nvidia Apex GitHub repo and paste the link of that issue here. So you and PyTorch team can refer that.
Once again thank you