I am running experiments on synthetic data (e.g. fitting a sine curve) and I get errors in pytorch that are really small. One if about 2.00e-7. I was reading about machine precision and it seems really close to the machine precision. How do I know if this is going to cause problems (or if perhaps it already has e.g. I can’t differentiate between the different errors since they are “machine zero”).
errors:
p = np.array([2.3078539778125768e-07,
               1.9997889411762922e-07,
               2.729681222011256e-07,
               3.2532371115080884e-07])

m = np.array([3.309504692539563e-07,
                 4.1058904888091606e-06,
                 6.8326703386053605e-06,
                 7.4616147721799645e-06])

what confuses me is that I tried adding what I thought was to small of a number so that it returned no difference but it did return a difference (i.e. I tried to do a+eps = a using eps = smaller than machine precision):
import torch

x1 = torch.tensor(1e-6)
x2 = torch.tensor(1e-7)
x3 = torch.tensor(1e-8)
x4 = torch.tensor(1e-9)

eps = torch.tensor(1e-11)

print(x1.dtype)
print(x1)
print(x1+eps)

print(x2)
print(x2+eps)

print(x3)
print(x3+eps)

print(x4)
print(x4+eps)

output:
torch.float32
tensor(1.0000e-06)
tensor(1.0000e-06)
tensor(1.0000e-07)
tensor(1.0001e-07)
tensor(1.0000e-08)
tensor(1.0010e-08)
tensor(1.0000e-09)
tensor(1.0100e-09)

I expected everything to be zero but it wasn’t. Can someone explain to me what is going on? If I am getting losses close to 1e-7 should I use double rather than float? googling it seems that single is the precision for float afaik.


  
      stackoverflow.com
  
  
      
    
  

  What is the machine precision in pytorch and when should one use doubles? 8



  pytorch



  asked by
  
  
    Charlie Parker
  
  on 07:52PM - 09 Sep 20 UTC


  
  
    
    
  
  



  
    
    
    How does one start using double without unexpected bugs? 
  
  
    On a related issue to this, I get errors in my regression that are close to machine precision (10^-7). As I tried things out machine precision things might be a lot more subtle than I expected, so I want to make sure. Is changing all the parameters to double in my model sufficient so that EVERYTHING is now computed using double to avoid errors? 
(btw read that post for more detail comments on issues if your curious) 
But what I want is: 

change code so everything uses double
hope that there are…