One more related question:
I have network M which predicts the parameters(weights and biases) for network K. So, here only network M’s parameters are learned whereas network K’s parameters are predicted from network M’s output. What is the efficient way to load the predicted parameters for network K(Note: network K parameters are not learnable)?
Currently, I do it like this. Below is the forward method of network K. Basically, I do a manual torch.matmul(). Is there any efficient way to just load the parameters and make prediction but not learn those parameters?
def forward(self, input, params=None):
   
        bias = params.get('bias', None)
        weight = params['weight']

        output = input.matmul(weight.permute(*[i for i in range(len(weight.shape)-2)], -1, -2))
        output += bias.unsqueeze(-2)
        return output