Heya,
the comment about claiming inventions was not “aimed” at anyone, I’m sorry if it came across awkward in your thread.
One quick comment:



 ChengzheXu:

W’ = sign( W )* mean(abs( W ))


when you have this in PyTorch (or in general), the gradient of sign will be 0 mostly - the sign function has derivative 0 except at 0. We can try this:
a = torch.randn(5, requires_grad=True)
b = a.abs().mean()*(torch.sign(a))
b.retain_grad()
b.sum().backward()
print ("b.grad", b.grad)
print ("a.grad", a.grad)

will give
b.grad tensor([ 1.,  1.,  1.,  1.,  1.])
a.grad tensor([ 0.2000,  0.2000, -0.2000, -0.2000,  0.2000])

or so which looks about right (depending on the ratio of positive vs. negative numbers in the random sample, the 0.2 (3-2) will be 0.6 (4-1) or 1.0 (5-0)).
I’m not sure I understand the update rule – maybe you want sign(W) in there?
But you could do W_prime.retain_grad() as above (again, I would not make  W’ a parameter) and then just do
with torch.no_grad():
    W.add_(-lr, torch.sign(W)*W_prime.grad)

or so.
Would that work as expected?
Best regards
Thomas