ok! seems like the entire drama was solved by changing the Relu at the end of the network to LeakyRelu.
i don’t fully understand what happened, maybe albanD can comment?
my intuition is that the entire network is composed of LeakyRelus and norm layers, so it must contain a lot of negative activations, such that it happens to add up as negative when entering relu - thus outputting a zero activation in the last neuron.
having a zero activation at the last neuron, according to my understanding, shouldn’t prevent the backprop from occurring normally. calculating dLoss/dLastNeuron doesn’t concern this zero value.
BUT, calculating the change in loss with respect to the fully connected layer, it might be trapped thanks to relu and a quite negative sum.
what i mean is that a small change in the activation of one of the neurons summing up to the relu, would not change the outcome of the relu at all because under a small change it would stay negative. and leakyRelu solved this issue.
can anyone comment on this?
thanks for helping everyone!