I am trying to train a few different types of multilayer networks. In each of these networks the weights and biases for each of the layers are not updating except for the weights and biases of the final layer and the normalization of the final ltsm layer.
I printed out the gradients for each layer and although for some layers its really small (10^-8 or 0), for other layers its only 10^-2 or 10^-1 and those weights also don’t update. I have ltsm layers. convolutional layers, batch and layer normalizations, and fully connected layers in these networks. I am using the relu activation function for all layers except for the ltsm layer where I am using the tanh activation. Additionally for each layer first the layer operation is done then the batch/layer normalization and then the activation function. How can I make sure that the weights update for the gradients that are not to small and how can I combat against really small weights? Also is there a way to programmatically check if the weights changed for each layer?