Hmm. Interesting. The code base has definitely changed a bit since I last dived into the convolution implementations about a year ago, but this THCUNN implementation 10 appears to use a for-loop. I am using the GPU backend.
I guess I’m also just surprised by how poorly this scales if it’s not sequential. When testing such that the actual spatial convolution is fixed (i.e. C x H x W doesn’t change). My network with N=1280 processes a batch in ~0.4 seconds on average. When N=20480, takes almost 5 seconds. That’s linear scaling, which seemed to suggested that these batches are being processed sequentially.