Hi,
Pytorch does not have a notion of “free parameters” (I am not 100% sure what you mean by that).
If you’re saying that the parameters of a layer are actually a simple transformation of a smaller set of parameters and that you want to optimize this smaller set of parameters, then you should write that explicity as a layer and it will work as you want:
class MyLayer(nn.Module):
    def __init__(self):
        super(MyLayer, self).__init__()
        # The parameters of your layer are just the "free parameters"
        self.free_params = nn.Parameters(small_params_size)

    def forward(self, input):
        # Expand your small parameters set into the full parameters
        full_params = expand_params(self.free_params)
        # Do the regular forward function
        output = layer_function(full_params, input)
        return output

If you use this layer, then all the regular pytorch methods will directly optimize the small set of parameters.