Hi everyone,
I am encountering a problem while training a pretty simple neural network where each layer is a triangular matrix weight matrix (alternating between lower and upper) and a SoftPlus activation applied at every layer, except the last layer. I was trying to see how the sum of the squares of each weight matrix gradient was varying over the number of iterations of training, and it was found to be pretty noisy with a large variance.
I was suspecting vanishing or exploding gradients so I used some solutions like gradient clipping using grad_clip_norm_ and grad_clip_value_ but of no use.  Also, I used batch normalization to solve the problem but get more noisy gradients.
Any suggestions would be appreciated.