I use forward mode differentiation

Pytorch does not have forward mode differentiation. You mean you use double backward trick to get a jacobian vector product?
I’m not sure to understand what you compare the runtime with? Can you give a full code sample?