Hi mbehzad!



 mbehzad:

is, isn’t it wrong in some sense?


Well, I suppose it depends on what your expectations are …
But you might wish to base your expectations on some other functions:
x**2 maps (-inf, inf) to [0.0, inf), but we don’t expect x**2 = x
to hold true for x >= 0.0, that is for values of x in [0.0, inf).
Or, back in the pytorch activation function world, torch.sigmoid() maps
(-inf, inf) to (0.0, 1.0), but torch.sigmoid (torch.sigmoid())
isn’t equal to torch.sigmoid().
Here’s another thing to consider:
softmax ([0.0 + delta, 1.0 - delta])
How would you like softmax() to behave when a negative delta
becomes zero and then crosses over to become positive?  Bear in
mind, you want this behavior to be usefully differentiable to support
backpropagation.
K. Frank