After laying down a lot of infrastructure (C++ frontend), I am just launching into attempting to train. I am working with a 4-class image classification problem, with several thousand 41X41 images. The model is a couple of chained convolutions with rectifiers in between, followed by a MaxPool and flattening to feed into a single Linear layer. (I know the model has not been “tuned” in any way, and is likely primed for serious overfitting, but this is just a shake-down cruise.)
When using Adam optimization, MSELoss (reduction=sum) and a moderate batch size, the total loss moves downward pretty nicely, with the expected amount of oscillations due to the stochastic nature of the optimization. When I inspect the activations in the Linear layer after a decent number of iterations, it shows values of both signs (not in itself concerning), with absolute values both fairly small and of order 1. In addition, there are a lot of extremely small values hovering right around ±10^(-42) or so, with exponents in that general range. (I am inspecting these values by converting values from the original Float32 tensors to double precision, through my own adaptor system.)
I was just wondering if, to those more experienced in this, this behavior raises any specific flags as a symptom of a specific issue. As I said, the overall progress of the optimization seems quite satisfactory. Is the smallness of these activations just an indicator that they are essentially “irrelevant”, or is there some possible numerical artifact at work?
Thanks,
Eric