I am sure that these methods cause the error, because this error is raised when I replace relu activations to this one.
There is even more. If I use this implementation of swish I run out of memory:
def swish(input):
    return input * torch.sigmoid(input)

But if I change it as follows everything fits into memory:
def swish(input, inplace = False):
input = input * torch.sigmoid_(input)
    return input

So I am looking for the ways to optimize tanh and softmax and make those more memory efficient.