I don’t know why the matrix values should be binary. You can apply a sequence of sigmoid -> minus 0.9 -> relu to obtain hard-binarized values in a learnable way, but the problem is that your model may not converge.
Any way, to follow the above process, I recommend you to register a tensor as parameters as follows:
class BinaryLayer(nn.Module):
    def __init__(self, in_channels, out_channels): 
        super(BinaryLayer, self).__init__() 
        self.weight = nn.Parameter(torch.randn(in_channels, out_channels), requires_grad=True)  #
    def forward(self, x): 
        bin_weight = torch.nn.functional.relu(torch.sigmoi(self.weight) - 0.9)
        y = torch.mm(x, bin_weight)
        return y

The maybe some error in the code snip, but  the flow is clear. After the training is complete, you can get a binary matrix in the same way the bin_weight is computed above.