I wanted to add one more thing I’ve tried. I replaced the forward part of the model with the following:
    def forward(self, padded_seq, lengths):
        
        # embedding layer
        x = self.embedding(padded_seq)
        x = pack_padded_sequence(x, lengths, batch_first=True)

        # lstm layer
        x, hidden = self.lstm(x)
        x, lengths = pad_packed_sequence(x, batch_first=True)
        
        # linear layer1
        out = self.dropout_1(x)
        out = self.fc1(out)   
        out = self.sigmoid(out)
        
        # mask outputs to those connected with the final hidden state
        return self.mask(out, lengths)
    
    def mask(self, tensor, lengths):
        idx = torch.tile(torch.arange(max(lengths)), (tensor.shape[0], 1))
        mask = idx == torch.unsqueeze(lengths - 1, axis = 1)
        return tensor[mask]

This resulted in the now all too familiar loss curves:

image723×265 15.3 KB

One thing it did help with is that it let me increase the batch size without getting those crazy loss curves. So that means I’m getting closer.