That’s linear scaling, which seemed to suggested that these batches are being processed sequentially.

Or that it is already compute bound   at N=1280
If you’re using the GPU backend in particular, check nvidia-smi to see what the utilisation is.
Also you are most likely not using the THCUNN code but cudnn.
If you have especially unusual sizes (compared to classical resnet-like CNN), try setting torch.backends.cudnn.benchmark = True. This will allow cudnn to test more algorithms (so the first time it sees a new size it will be slower, but subsequent runs should be faster).