Yes you are right, I am doing deep supervision. I have 5 outputs and some of the outputs are -inf, NaNs and some real values. The final loss is a sum of all losses so that too goes to NaN. How can I fix this? This only happens when continuing the training of my good model.
I am suprised that all are not NaNs. There are some valid outputs. I was suspecting it is due to a batchnorm layer having too low std but am unsure now. Any suggestions to debug?