The transfer time is defined by e.g. the connection of the GPU to the host.
For your particular issue, make sure you are timing the operations correctly. Since CUDA operations are executed asynchronously, you would have to synchronize the code before starting and stopping the timer.
If you don’t synchronize, you might be profiling the kernel launch (not the forward pass), while the transfer to the host would automatically synchronize the code and accumulate the forward pass + the transfer time.