silvester:

self.weights = weights* nn.Parameter(torch.ones(1))


This creates untracked parameter - it is not in module, but in backward graph. Optimizer doesn’t process it, so you get error on second iteration.
self.weights = Variable(self.weights.data, requires_grad=True) 

this probably just makes first assignment have no effect, and is equivalent to
self.weights = nn.Parameter(torch.ones(1) * weights)

or torch.full((1,),weights)



 silvester:

if you look at this post, it is suggested to do multiplication for the trainable variable.


list(LearnedSwish().parameters())
[]

good luck optimizing that