Hello, @Oktai15. I also came across a similar problem while training a segmentation network using semantic-segmentation-pytorch 12. After running for some random number of epochs, all volatile GPU util becomes 100% and the training seems to be stuck.
Did you fix the problem or have any thoughts about it?