Your function looks correct.
Some suggestions:

You dont need to save the entire mat for backwards, only mat.size()


mat.size()[0] -> mat.size(0)

you dont handle trace for non-square inputs.
the output return type is not the same as the input return type, because you use torch.Tensor. Instead use mat.new()


Overall, here’s a modified version that handles non-square inputs, and implements the other suggestions.
class CustomTrace(torch.autograd.Function):
    
    def forward(self, input):
        self.isize = input.size()
        return input.new([torch.trace(input)])
    
    def backward(self, grad_output):
        isize = self.isize
        grad_input = grad_output.new(isize).copy_(torch.eye(*isize))
        grad_input.mul_(grad_output[0])
        return grad_input