Update:
I trained the same data in ResNet152:
ResNet152 on 4GPU finetune:
Training complete in 64m 55s
Best val Acc: 0.987468
ResNet152 on 4GPU freeze:
Training complete in 59m 18s
Best val Acc: 0.727110
It’s growing the number of GPUs what is making it slow, so the data IO is significant. I wonder what would be with a bigger dataset