Hi Dan!



 Harimus:

eps is supposed to be the smallest number where 1.0 + eps != 1.0


This is a documentation bug.  Quoting from pytorch’s 1.8.1 documentation:
eps     float     The smallest representable number
                  such that 1.0 + eps != 1.0.

This is sort of on the right track, but really isn’t correct.
Numpy’s (1.20) documentation gets it right:
eps : float

    The difference between 1.0 and the next smallest
    representable float larger than 1.0. For example,
    for 64-bit binary floats in the IEEE-754 standard,
    eps = 2**-52, approximately 2.22e-16.

What’s at issue in your numerical test is that neither 1.0 + 0.5 * eps
nor 1.0 + 0.9 * eps is exactly representable, so the floating-point
arithmetic has to decide whether to round up or down to a representable
number.
In the first case, it rounds down to 1.0; in the second, up to 1.0 + eps.
(By definition, there are no representable numbers in between.)
Here’s an illustration that shows what is going on including the correctness
of numpy’s definition of eps:
>>> torch.__version__
'1.7.1'
>>> torch.tensor ([1.0]) + 0.5000001 * torch.finfo().eps == 1.0
tensor([False])
>>> torch.tensor ([1.0]) + 0.5000000 * torch.finfo().eps == 1.0
tensor([True])
>>> (torch.tensor ([1.0]) + 0.5000001 * torch.finfo().eps) - 1.0 == torch.finfo().eps
tensor([True])
>>> (torch.tensor ([1.0]) + 0.5000000 * torch.finfo().eps) - 1.0 == torch.finfo().eps
tensor([False])

Best.
K. Frank