Solved by greaber in post #2 
              
              
                I find it simplest to use activation functions in a functional way.  Then the code can be 
def swish(x):
    return x * F.sigmoid(x)