As far as I understand your issue, the training script takes 16GB at most running on the GPU and more than 128GB on the CPU?
If that’s correct, do you see an increasing memory usage during training or does your script run out of memory during the first iteration?
Did you change something in your data loading pipeline, e.g. are you loading the complete dataset into RAM?