That is indeed interesting.
I did a quick test and it has nothing to do with the backwards: if you only measure the forward time, you see the same behavior 
So I would guess that the kernel for larger batch size are more optimized than the ones for small batch size. And this is why you see this.
Note that if you use cudnn, it might even take completely different algorithms depending on the input shape!