Thanks for the tip, my code was running using a multi-gpu setup so initially CUDA_LAUNCH_BLOCKING=1 was not possible. I reran it using a single GPU with profiling and this the result. This is more sensible now, but in the interest of optimization to the max, I wonder if there’s anything more that can be done? 
Discriminator2.png1051×991 125 KB