How are you profiling the code?
Since CUDA operations are executed asynchronously, you would have to synchronize the code before starting and stopping the timers.
Otherwise your profiles will be wrong and would accumulate the runtime of previous kernels into a blocking operation.