Ok let’s see if this is a good summary of what I think is correct (modulo ignoring some details that I don’t fully understand right now of floats, like the bias).
But I’ve concluded that the best thing for me is to make sure my errors/numbers have two properties:



they are within 7decimals of each other (due to the mantissa being 24 bigs like you pointed out the log_10(2^24) = 7.225)




they are far enough from the edges. For this I take the mantissa to be 23 bits away from the lower edge (point position about -128+23) and the same for the largest edge but 127-23.



As long we satisfy that more or less we avoid adding two numbers that are too small for the machine to distinguish (condition 1) and avoid overflows/underflows (condition 2).
Perhaps there is a small detail I might be missing with the bias or some other float detail (like representing infinity, NaN). But I believe that is correct.
If anyone can correct the details, that would be fantastic.

my current conclusion is because most learning rates are  1e-1 ish I will have everything be 7 decimals from zero. That seems the safest. As lost as every number is in the range of 1.0 ± 7 decimals it should be fine. This really guarantees there won’t be weird machine precision if we stick to an absolute scale rather than a fixed one.