Solved by ptrblck in post #4 
              
              
                No, you would have to implement the backward method manually as shown in the linked example. 
E.g. for ReLU: 
    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and…