Are you using different CUDA streams for every thread?
By default they will all use the same CUDA stream, which will serialize all operations.
See c10::Stream 5 if you’re developing against master, or this one 5 if you’re developing on 1.0.1.