can i see your benchmark script?
for batch size 42, the difference in memory usage is minor, 6907 (pytorch) vs 7007 (Caffe). I expect that for batch size 48 it tips in either direction someway (pytorch might be picking a different algorithm for cudnn probably.