Hi @ptrblck
thanks for the response, the example you gave can only work if you define ‘w1’ as Parameter, lets see if this is not:
import torch
import torch.nn as nn
from collections import OrderedDict

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.w0 = nn.Parameter(torch.randn(1))

    def forward(self, x):
        return x + self.w0

model = MyModel()
print(model.state_dict())
OrderedDict([('w0', tensor([0.4342]))])

delattr(model, 'w0')
print(model.state_dict())
> OrderedDict()

setattr(model, 'w0', torch.randn(1)) #nn.Parameter(torch.randn(1)))
print(model.state_dict())
> OrderedDict()

Basically, here is what I need to do, I freeze each parameter of the network, then I need to project it to a random space, like
theta = theta + PX
then X is the shared parameter between all parameters of the network and the only one which gets updated and theta is the initial value for the parameter.