Thanks for the reply @tom
yea, the indent for the forward function got messed up while I was pasting the code.
“You could use self.log_alpha, log_delta, log_r as the parameter (but ideally init to something close to in 0 instead of 1, too) and then do alpha = self.log_alpha.exp().expand_as(x).”
I am confused as to what you mean exactly. Lets say I init them properly. in the paper they initialize with a normal distribution with mean 1 and std 0.1.
Q. When exactly would I take the log?


I thought I could do something like - for a simple version of SGD, though it would be nice to use pytorchs optimizers
for p in pcen.parameters():
p_log = torch.log§
p_log.data.add_(-learning_rate, p.grad.data) #or p_log.grad.data?
# and then somehow copy the exponentiated log parameters back to p


Q. or is all this not necessary based on the approach you proposed?
Thanks,
Gautam