Can the parameters in the model after certain operation cause the gradient to explode that cause overflow ? Or i mainly should debug the input ?
Now there is no NaNs in the output of each layer , however still the loss scaling is decreasing.