Thanks for the detailed explanation and I think your “sanity check” makes sense.
Note that I don’t have a background in numerical mathematics, so please take this post with a grain of salt.



 ericrhenry:

At the end of the optimization run, I am just inspecting the values directly using the tensors returned by the model->parameters() call. I had earlier questions about the “custody” of these values (presumably maintained via shared_ptr/intrusive_ptr semantics), that they are in fact a direct reflection of the values currently being held by the optimizer. They apparently evolve from the originally random values assigned when the model is “registered”, which suggests this is the case.


Yes, this is correct. The optimizer stores references to all parameters and you could inspect the id() of the parameter inside the model, which should match the stored references inside the optimizer.



 ericrhenry:

Secondly, the set of values I refer to are well separated from the others in log space and are often contiguous within the parameters tensor for the activations. Given that the values are supposedly maintained as Float32, they all fall in absolute value just below the nominal “smallest” such value (1.2e-38). If I let Pytorch just print out the tensor itself (i.e. without conversion to my double-precision space), the activations are identified as “CPUFloatType”, but it actually prints out values such as 3.7737e-42, which simply do not fall within the “usual” range of possible values. They do however fall in the range of denormalized values (where the mantissa progressively loses precision), which extend down to about 1.e-45.


Would this not be expected assuming the weight should be updated to a zero value?
During training your optimizer might scale the learning rate (e.g. Adam uses internal running estimates), so that further updates might be smaller in their magnitude. Could this explain the small denormal values or what is the concern about the denormals?



 ericrhenry:

I am using Adam optimization, and when I use the default (0) value for the weight_decay parameter, absolutely none of these extreme values appear; in fact, the dynamic range of activations appears to be no more than a factor of 1000 or so. My original observations were based on assigning a value of 0.01 to this parameter, which produces these odd FP values.


Weight decay would add the norm of all parameters to the loss, such that the magnitude of these parameters would be limited (or converge towards zero).
Let me know, what you think and what your concerns are.