albanD:

That would explain why the throughput looks this way.


Are you sure about that?  In a linear scaling, one expects what I call sample rate above (batch_size / (t1-t0)) to saturate.  However, it starts to decrease with increasing batch sizes (see post above).  Larger batches, it seems, are computed less efficiently than running multiple computes on smaller batches.