CUDA operations are asynchronous, so if you want to measure the forward pass you would have to synchronize the code before starting and stopping a timer.
Currently you might be profiling the PyTorch overhead and the kernel launches.