Thanks for the simple example @ ptrblck can I also do this in the forward function itself such that it yields the same results?
That is:
if rand > 0.5:
      self.weight = nn.Parameter(torch.cat((self.weight, torch.randn(10, 1)), 1))