I am trying to implement a model for prob.density estimation that is going to map a particular distribution into normal distribution. So my loss is somewhat a negative log-likelihood function which I am trying to minimize.
My custom forward function for every layer looks something like:
def forward(self, input): 
.    self.Y_l = input.mm(self.W)
.    self.Z_l = self.phi(self.Y_l)
Each self.phi is a nn.Softplus() function
Now I basically have 5 or 7 such layers in the network and I try to minimise some log likelihood loss based on self.W (which is non-convex I think) using torch.optim.adam as the optimizer. I try to observe the mean of self.W.grad.abs( ) for every layer as training progresses and it shows a noisy curve which I trying to figure out why.