I tried with num_workers=0 or other non-zero values, seems no help on this issue.
Of course I used the same batch size, same training dataset, same seeds, only network changed.
And I also tried on another PC, the issue also exist.
it’s really weird…