As a first pass response you can use the Module class for your 1st and 3rd layers.  As I understand these layers use the same weight matrix but transposed. This is an interesting implementation, but to ensure that the parameter updates are correct, you should actually have one self.weight matrix that is applied twice within the module, either in the original or transponsed form. You would still need two bias vectors
So the forward pass would be like:
def forward (input, first=False):
    if first:
        out = torch.matmul(input, self.w) + self.b_1
    else:
        out = torch.matmul(input, self.w.T) + self.b_2

Where self.w is a parameter layer of shape (n1, n2) and self.b_1 has shape (n1) and self.b_2 has shape (n2).