Hi Steve!



 stevethesteve:

def log_softplus(x):
  if x > -5:
    return torch.nn.functional.softplus(x).log()
  else:
    return x

which is numerically stable for small x. If done naively (softplus then log), softplus would return a very small number with low precision.


I think your concern about loss of precision is a partial red herring.
Because we’re working with floating-point numbers, we don’t lose
precision when softplus (x) becomes small (say, on the order of
10**-7).  We only start to lose precision when softplus (x) becomes
very small and starts to denormalize and then underflows to zero
(at which point the subsequent log() will return -inf). This starts to
happen around 10**-38 (which corresponds to an x of about -90.0).
Whether this matters depends on your specific use case.  But, unless
your argument to log-softplus can reasonably become as negative as,
say, -90, your “naive” implementation will be fine.
On the other hand, if your argument to log-softplus can reasonably
become that negative, then your suggested version does make
sense, except that you should use a 'breakpoint" (x > -5) that is
significantly more negative than -5 (perhaps something like -40).
As it stands, your breakpoint of -5 does more harm than good,
because in this range x is an imperfect approximation to log-softplus.
The following illustrative script uses a double-precision log-softplus
computation as a surrogate for the exact result:
import torch
print (torch.__version__)

def log_softplusA (x):
    return torch.nn.functional.softplus (x).log()

def log_softplusB (x, brk = -5.0):
    return torch.where (x > brk, torch.nn.functional.softplus (x).log(), x)

x = torch.arange (-100, 101, 1).double() / 10.0

lsp = log_softplusA (x)   # double-precision "ground truth"

diffA = log_softplusA (x.float()) - lsp
diffB = log_softplusB (x.float()) - lsp

print ('diffA.abs().max() =', diffA.abs().max())
print ('diffB.abs().max() =', diffB.abs().max())
print ('diffB.abs().argmax() =', diffB.abs().argmax())
print ('x[diffB.abs().argmax()] =', x[diffB.abs().argmax()])

print ('log_softplusA (torch.tensor ([-90.0])) =', log_softplusA (torch.tensor ([-90.0])))
print ('log_softplusA (torch.tensor ([-100.0])) =', log_softplusA (torch.tensor ([-100.0])))
print ('log_softplusA (torch.tensor ([-110.0])) =', log_softplusA (torch.tensor ([-110.0])))
print ('log_softplusA (torch.tensor ([-110.0], dtype = torch.double)) =', log_softplusA (torch.tensor ([-110.0], dtype = torch.double)))

And here is its result:
1.7.1
diffA.abs().max() = tensor(8.1795e-07, dtype=torch.float64)
diffB.abs().max() = tensor(0.0034, dtype=torch.float64)
diffB.abs().argmax() = tensor(50)
x[diffB.abs().argmax()] = tensor(-5., dtype=torch.float64)
log_softplusA (torch.tensor ([-90.0])) = tensor([-90.])
log_softplusA (torch.tensor ([-100.0])) = tensor([-99.9831])
log_softplusA (torch.tensor ([-110.0])) = tensor([-inf])
log_softplusA (torch.tensor ([-110.0], dtype = torch.double)) = tensor([-110.], dtype=torch.float64)

Best.
K. Frank