are you profiling correctly?
If you are using python timing calls, you have to insert cuda synchronize calls to get correct timing:
torch.cuda.synchronize()
now = time.time()
# my cuda calls
torch.cuda.synchronize()
total_time = time.time() - now