Thanks again @albanD . I see the output of print(model.mu.grad) is in range tensor([[ 5.7729e-07], ... These values are too low to reflect in an update, right? Is this do do with dynamic range of my initialization or that of inputs?