Values being zero is actually what is confusing indeed, my wild guess is that you’re doing some kind of batch normalization or L2loss (weight decay) that might keep the values of the other outputs closer to zero, this is just a guess, I am not completely sure what’s happening, what you could do is try and debug a sample from one layer to the other and see what’s actually going on