Another workaround is to create custom sign() function that does not modify incoming gradients.
class SignPassGrad(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        return input.sign()

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output

class BinaryLinear(nn.Linear):
    def forward(self, x):
        return F.linear(x, SignPassGrad.apply(self.weight), self.bias)

And it does not yield any speedup (probably due to fast memory copies in torch.clone).