It’s weird. I found out it seems like the Gpu cache release problem of pytorch. I add ‘torch.cuda.empty_cached’ in somewhere of my code and every gpu have same memory usage. But the program runs rather slower since the empty_cached was add in a for loop.
I still cannot found out what’s wrong. Does it in theory, if you use distributed training, every gpu will have the same memory usage? I know that if you use dataparallel module, the gpu0 will have more memory consumtion.