Also this is a modified version of your code that prints runtime instead of throughput.
https://colab.research.google.com/drive/1aMCMZaSBak0PssL8nt2ff3mgFRKeNWaP 1
This makes it clear what happens: for small enough batch size, you get a sub-linear scaling as the GPU is under-utilized. At some point, you fully use the GPU and you start scaling linearly.
That would explain why the throughput looks this way.