I’m still unsure how the code should work.
You are currently creating W0 with two values. Is this your use case or would you like to create a parameter with the shapes [self.feature_len, self.z_dim]? In the latter case you would need to use a factory method such as torch.zeros, torch.randn etc.
You should also register the parameter via self.W0 = nn.Parameter(...), so that it’ll be returned in model.parameters(), which is the usual way to forward the parameters to an optimizer.
Once the code is running, you should check, if model.W0.grad gives you a valid gradient, as I’m currently unsure if the reassignments detach the computation graph.