@ptrblck No i didn’t run stress test yet.
I have noticed another thing. I am using num_workers = 5 in the DataLoader. So htop shows 6 processes in total for this job, (which means 5 for DataLoader workers plus 1 main). When the training stucks (slows down), 5 out of 6 processes go to D state (i.e., uninterruptible sleep) and they can’t go out of this state anymore. Whereas in the beginning (normal training) they keep fluctuating between D and R states.