malbergo:

do you know of any activation function whose inverse has output range (0, inf)


I know this is a bit necro, but … wouldnt a function whose inverse has output range (0, inf) mean that any input value less than 0 would be illegal? Wouldnt that destabilize the network some?
Alternatively, I guess taking the square sort of complies with this, if you always implicitly take the positive square root in the inverse direction, perhaps?