I benchmarked against the same number of elements but arranged as a B x C X H x W tensor with N evenly distributed among H x W and it takes about 0.3 seconds per batch with N=20480. I don’t think it’s a compute issue. Even if batches are parallelized, I think it must be to a lesser extent (or have inherently less gain from parallelization) than the spatial convolutions.