Hi everyone,
I have been reading articles on parse Biaffine for several days, but I just can’t understand how the biaffine module should be implemented. My parser uses an affine MLP and I just can’t find how to change it to Biaffine.
The structure of my parser is as follows:
Parser(
  (dropout): Dropout(p=0.6, inplace=False)
  (word_embedding): Embedding(381, 100, padding_idx=0)
  (tag_embedding): Embedding(17, 40, padding_idx=0)
  (bilstm): LSTM(908, 600, num_layers=3, batch_first=True, dropout=0.3, bidirectional=True)
  (bilstm_to_hidden1): Linear(in_features=1200, out_features=500, bias=True)
  (hidden1_to_hidden2): Linear(in_features=500, out_features=150, bias=True)
  (hidden2_to_pos): Linear(in_features=150, out_features=45, bias=True)
  (hidden2_to_dep): Linear(in_features=150, out_features=41, bias=True)
)

where Embedding = nn.Embedding and Linear = nn.Linear.
Now, what I need to change to implement BiAffine layer?
I think that this two layers:
self.bilstm_to_hidden1 = nn.Linear(...)
self.hidden1_to_hidden2 = nn.Linear(...)

are simply MLPs in which there is only one Linear layer.
So I think the right thing to do is replace
self.hidden2_to_pos = nn.Linear(...)
self.hidden2_to_dep = nn.Linear(...)

with something a BiAffine module. What I found as a ““standard”” implementation in pytorch is the following:
class BiAffine(nn.Module):
    """Biaffine attention layer."""
    def __init__(self, input_dim, output_dim):
        super(BiAffine, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.U = nn.Parameter(torch.FloatTensor(output_dim, input_dim, input_dim))
        nn.init.xavier_uniform(self.U)

    def forward(self, Rh, Rd):
        Rh = Rh.unsqueeze(1)
        Rd = Rd.unsqueeze(1)
        S = Rh @ self.U @ Rd.transpose(-1, -2)
        return S.squeeze(1)

but it doesn’t seem to work.
What am I doing wrong?
Do you have any ideas / suggestions?
Do you know if there is any code / module that I can use as a baseline to implement this module?
I no longer know where to turn.
Thanks a lot to everyone.