Please, advise how to make this work.
class HalfReLU(nn.Module):
	def __init__(self):
		super(HalfReLU, self).__init__()

	def forward(self, x):
		half = x.shape[1]//2
		x[:, :half] = nn.functional.relu(x[:, :half], inplace=False)
		return x

I’m getting

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 128, 8, 8]], which is output 0 of SliceBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Whether relu is inplace or not doesn’t matter. Works (or just doesn’t give errors, don’t know for sure) in case of things like
		x[:, :half] *= 2

and
		x[:, :half] = nn.functional.sigmoid(x[:, :half])

but not in case of relu. I’m guessing it’s sort of a bug/edge case? For leaky_relu also doesn’t work.