Hello Kejie!



 Kejie_Tang:

I want to define the activation function ReLU(x) * ReLU(1-x).


The short answer is that you just do.
torch.nn.functional.relu() (and its class version, torch.nn.ReLU)
is differentiable (in the pytorch sense), so its product is as well, and both
relu() and its product work just fine with autograd and backward().
I don’t understand the point of the code you posted, nor its relevance
to the question in the title of your post, but, quite simply:
import torch
print (torch.__version__)
one = torch.autograd.Variable (torch.FloatTensor ([1.0]))
x = one / 2
torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
x = -one
torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
x = 10 * one
torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)

(The nonsense with autograd.Variable is because I’m using pytorch
version 0.3.0.)
Here are the results:
>>> import torch
>>> print (torch.__version__)
0.3.0b0+591e73e
>>> one = torch.autograd.Variable (torch.FloatTensor ([1.0]))
>>> x = one / 2
>>> torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
Variable containing:
 0.2500
[torch.FloatTensor of size 1]

>>> x = -one
>>> torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
Variable containing:
 0
[torch.FloatTensor of size 1]

>>> x = 10 * one
>>> torch.nn.functional.relu (x) * torch.nn.functional.relu (1 - x)
Variable containing:
 0
[torch.FloatTensor of size 1]

Best.
K. Frank