It depends on your GPU and GPU, likely. In my experience, there is a speedup to be had with the GPU even in batch 1 resnet18, but again, I am sure there are combinations of GPU and CPU where it comes out differently.
Best regards
Thomas