What’s the difference between nn.ReLU() and nn.ReLU(inplace=True)?
yunjey
apaszke
laoreja
is not an in place operation, because you use the same variable name, but it’s not the same variable underneath. You just point your x name to a new variable, the old one is still in memory (because it’s referenced by the pytorch graph)
cdancette
laoreja
I’m a newbie of pytorch. So I wonder whether nn.ReLI(inplace=True) would do harm to backprop? And what about F.ReLU(inplace)?
harryhan618
rasbt
yxchng
For relu, when input is negative, both the grad and output should be zero, grads should stop propagating from there, so inplace doesn’t hurt anything while saves memory.
streaver91
Is this an in-place operation? Thanks!
Jay_Timbadia
cdancette
You can check on pytorch: <SCODE>>>> a = torch.tensor(1.0)
>>> torch.sigmoid(a)
tensor(0.7311)
>>> print(a)
tensor(1.0)
<ECODE> <SCODE>>>> a = torch.tensor(1.0)
>>> torch.sigmoid_(a)
tensor(0.7311)
>>> print(a)
tensor(0.7311)
<ECODE> In the first case, a still has its original value, while in the second case, a is different.
cdancette
In case y = F.relu(x, inplace=True), it won’t hurt anything if value of x should  always be positive in your computational graph. However, some other node that shares x as input while it requires x has both positive and negative value, then your network may malfunction. For example, in the following situation, If (1) is declared first and execuated first, then value of x is changed, then z may have incorrect expected value.
helson73
Hi all, Even though there are multiple answers, I will explain my problem here. I am facing the following error message: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4, 64, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck! <SCODE>    def forward(self, z, xlr=None, logdet=0, logpz=0, eps=None, reverse=False,
                use_stored=False):

        self.h_new, self.c_next = self.conv_lstm(z, (self.h, self.c))

        # Encode
        if not reverse:
            for i in range(self.L):
                print("Level", i)
                for layer in self.level_modules[i]:

                    if isinstance(layer, modules.Squeeze):
                        z = layer(z, reverse=False)
                        self.h_new = layer(self.h_new, reverse=False)

                    elif isinstance(layer, FlowStep):
                        z, logdet = layer(z, lr_feat_map=self.h_new, # lr_downsampled_feats[i + 1], # TODO: change this part
                                          x_lr=xlr, logdet=logdet, reverse=False)

                    elif isinstance(layer, modules.GaussianPrior):
                        z, logdet, logpz = layer(z, logdet=logdet, logpz=logpz,
                                                 lr_feat_map=self.h_new, #lr_downsampled_feats[i + 1],
                                                 eps=eps, reverse=False)
                        self.h = self.last_squeezer(self.h_new, reverse=True)
                        self.c = self.c_next
<ECODE> Do you think it would be the best to just pass on the hidden and context states through the function outputs? I have already set loss.mean().backward(retain_graph=True) and skimmed the code for other inplace operations. Any help would be much appreciated !! Please let me know if further code snippets are required.
cwinkler