Hi! I’m playing with custom activations and I used the following function instead of ReLU. After doing this I’m running out of GPU memory on Cuda.
def custom_activation(x):
    return x * tanh(softplus(x)) = x * tanh(ln(1 + exp(x)))

How can I make this code more memory efficient? May be you can recommend some in-place operations which I can use here?
Or may be there are more efficient ways to implement custom activations? Or may be there are some ways to manage caching on GPU?