You would have to synchronize the code before starting and stopping all timers, while this is not the case for the first call into the LSTM.
Also, I would recommend to use multiple iterations and calculate the average of the operations, as well as to add warmup iterations.
The torch.utils.benchmark utility does this automatically for you and is the recommended way to profile workloads.