I can’t imaging we’re at the compute bounds though. I’d thought of that, but H=W=4 in those tests and C scales by powers of 2. Effectively, I am running this with data equivalent to input image sizes of 3x256x512 in batches of 4 on each GPU. GPU utilization is at 100% though… I honestly don’t understand why this should throttle stuff so badly. I’m using a GTX 1080T and I’ve historically had better performances than this with larger image sizes.
I will try cudnn benchmark. Maybe that could help.