xinsss:

For example, the batch_size =16, the model is only using 8 in training.
When the model is in eval mode, I also tried to print the input tensor inside the model’s forward function. Sometimes it will print twice (each with batch of 8) as expected.
Any ideas?


I think may be because of the nn.DataParallel it got split into two GPUS 8 in each