In finetuning example, tensorflow batch size is more double than pytorch with resnet152 model and same machine.
I don’t exactly know but I saw pytorch error’s occuring in “input.new()” in batchnorm, so I guess two code treats different intermediate buffer with different ways.
if tensorflow use cpu memory to allocate intermediate varaible, goodness would be memory efficiency, but badness would be speed down by frequent cuda memory copy.