@smth I continue to work on my code and try to solve the problem of nan/inf batch loss. I found that the problem is related to the num_workers parameter in the torch.utils.data.DataLoader.
I have got an intel i7-6700HQ CPU with 4 cores and 8 threads. If I give a bigger value for num_workers eg. 6 or 7, there are more chances to have inf and nan batch loss. In contrast, if a value less than 4 is given, the inf or nan will never appear. I’m wondering if a bigger value of num_workers will create some conflicts.
In the other hand, I tried the num_workers=4 and num_workers=1 but I found almost no acceleration of speed. I found there seems no multi-processing according to my system resource monitor.
I would like to know if I am using this parameter correctly. My code is written in my first post. Thanks.