Thank you for your answer. Is there anything wrong with using Apex?
Why do you recommend me to use the native mixed-precision implementation?
Actually, I am working with 3d images that are high dimensional. So, I frequently encounter the OOM error.
By using apex everything was fine until some moments ago. But, again after some changes in my code I do not know why I am getting OOM error!
My second question is that, if mixed precision and DistributedDataParallel, can be used at the same time and using both of them can solve the problem of OOM error for larger batches? I tried DataParallel before and it didn’t help.
At the moment, I am working with batches of size 2, but in this batch size I do not observe good behavior of loss function and the loss function dose not decrease. It increase actually. After googling a lot I found that it’s due to batch size and dateset that I have. I tested it for 2D and it worked fine but for 3D I didn’t get good result.