You can set parameters not needing gradients to not require grad (using p.requires_grad_(False)). PyTorch only computes gradients required for those values that require gradients.
Even so, backpropagation has it that you use the chain rule from the final result (i.e. loss) back to the first use of the parameter that requires gradients. You cannot escape calculating gradients for the intermediates in between.

Best regards
Thomas