Solved by albanD in post #2 
              
              
                Hi, 
Weight decay is an l2 penalty on the loss function. 
So when you take the derivative, it becomes just the value of the weights (times 2). 
So we add that directly to the gradients.