How did you profile the code?
Note that CUDA operations are executed asynchronously, so I assume you’ve properly synchronized the code and isolated the bottleneck to the dropout layer or have used other profiling tools such as nsys?