Thanks for the response. I would like to add that you have been an absolute stalwart in addressing questions “all over the place” in these forums; just know that this is noted and very much appreciated.
Excuse the long answer; I just wanted to articulate why I thought a “sanity check” was warranted.
Source of the concern is rooted in over forty years dealing with numerical mathematics, which I guess fuels a general skeptical attitude. My first concern was about the validity of the source for these numbers: At the end of the optimization run, I am just inspecting the values directly using the tensors returned by the model->parameters() call. I had earlier questions about the “custody” of these values (presumably maintained via shared_ptr/intrusive_ptr semantics), that they are in fact a direct reflection of the values currently being held by the optimizer. They apparently evolve from the originally random values assigned when the model is “registered”, which suggests this is the case.
Secondly, the set of values I refer to are well separated from the others in log space and are often contiguous within the parameters tensor for the activations. Given that the values are supposedly maintained as Float32, they all fall in absolute value just below the nominal “smallest” such value (1.2e-38).  If I let Pytorch just print out the tensor itself (i.e. without conversion to my double-precision space), the activations are identified as “CPUFloatType”, but it actually prints out values such as 3.7737e-42, which simply do not fall within the “usual” range of possible values. They do however fall in the range of denormalized values (where the mantissa progressively loses precision), which extend down to about 1.e-45.
I also made a rather interesting/weird observation: I am using Adam optimization, and when I use the default (0) value for the weight_decay parameter, absolutely none of these extreme values appear; in fact, the dynamic range of activations appears to be no more than a factor of 1000 or so. My original observations were based on assigning a value of 0.01 to this parameter, which produces these odd FP values. (The two optimizations progress in very similar fashion.) I’ve now seen this consistent and distinct behavior toggling back and forth between the two values of this parameter several times. It’s likely that this behavior is readily explained in terms of how the Adam algorithm uses this parameter, and it’s possible that my choice of non-zero value is somehow inappropriate.
All part of my ongoing education process…
Thanks again,
Eric