I haven’t fully understood it yet,

What is unclear? The main idea is that if some input is not used in the output, then the gradient of the output wrt that input is 0.

PyTorch can’t keep up.

Yes unfortunately, pytorch was built for large GPU training and experimentation than CPU inference for small inputs (batch size 1).
The way we are going to make such inference more efficient is with the jit module and TorchScript.
Also if you are doing only inference, you should run your loop in a with torch.no_grad(): wrapper to avoid the autograd overhead.