This can happen when there is a lot of CPU overhead in some other part of the code relative to the amount of GPU computation. Can you post a runnable code snippet (e.g., something that corresponds to what you think the slowest part of the code) so we can reproduce and diagnose the issue?