I don’t think you actually can do that with Tensors, which is the whole point of torch.nn.Parameter.
My understanding is that Parameter was added specifically to avoid computing gradients for normal Tensors.
What is your use case for Tensors which are not part of the model?