I’m working with a segmentation-classification problem statement, where I need to split some text into segments based on its emotions. I have two outputs here: a binary output denoting whether the current sentence is a segment boundary or not, and a multi-label sigmoid output that produces the list of emotions/tags that apply for the latest segment.
I’m trying to partially replicate the model given in the following URL:

  
      web2py.iiit.ac.in
  
  
    
inproceedings.pdf.9272879615b02f00.417474656e74696f6e2d6261736564204e657572616c2054657874205365676d656e746174696f6e2028454349522d32303138292e706466.pdf 2

625.05 KB

  
  
    
    
  
  


The input is fed as a sliding window of size 7, with left and right contexts of a width of 3 sentences each. The central sentence is to be identified as a segment boundary.
The input sizes are:
torch.Size([3, 50, 300]) torch.Size([1, 50, 300]) torch.Size([3, 50, 300])

Since I’m feeding the elements individually, the batch size is 1. However, due to the input specifications of the CNN and LSTM layers, I am unable to correctly identify how I should feed the input.
Problem 1:
In the CNN layer, the input must be a 4D array of size [batch_size, num_channels, height, width]. I therefore reshaped the arrays to sizes [3, 1, 50, 300], [1, 1, 50, 300] and [3, 1, 50, 300].
Problem 2:
The LSTM layer requires the shape [sequence_len, batch_size, input_size]. In a step before this, I had concatenated the previous output to give a shape of [7,72]. I unsqueezed this output to get a shape [7,1,72] and fed it that.
Here’s the model for reference:
class ContextEncoder(nn.Module):
    def __init__(self):
        super(ContextEncoder, self).__init__()
        
        # CNN filters for context
        self.cnn_k2 = nn.Conv2d(1,2,kernel_size=(2,300))
        self.cnn_k3 = nn.Conv2d(1,2,kernel_size=(3,300))
        self.cnn_k4 = nn.Conv2d(1,2,kernel_size=(4,300))
        
    def forward(self, x):
        x = torch.stack([torch.squeeze(self.cnn_k2(x), 3), torch.squeeze(self.cnn_k2(x), 3), torch.squeeze(self.cnn_k2(x), 3)])
        x = F.max_pool2d(x, 2)
        return x.view(x.shape[1], -1)

class CNN_LSTM(nn.Module):
    def __init__(self, context_size):
        super(CNN_LSTM, self).__init__()
        
        # Context encoder
        self.context_encoder_lr = ContextEncoder()
        self.context_encoder_c  = ContextEncoder()
        
        # Hidden state generation
        self.lstm = nn.LSTM(72, 36)
        
        # Attention generation
        self.attention = nn.Linear(36, 36)
        
        # Output layers
        self.emotions = nn.Linear((2 * context_size + 1) * 36, 14)
        self.segment = nn.Linear((2 * context_size + 1) * 36, 2)
        
        # Initialize hidden state
        self.reset_hidden()
    
    def reset_hidden(self):
        """Reset hidden state with (h_n, c_n)"""
        self.hidden = (torch.zeros(1,1,36), torch.zeros(1,1,36))
        
    def forward(self, x_left, x_center, x_right):
        # Encode context
        c_l = self.context_encoder_lr(x_left)
        c_c = self.context_encoder_c(x_center)
        c_r = self.context_encoder_lr(x_right)
        
        # Stack contexts and unsqueeze batch size
        c = torch.cat([c_l, c_c, c_r])
        c = c.unsqueeze(1)
        
        # Pass context through LSTM
        x, self.hidden = self.lstm(c, self.hidden)
        
        # Generate attention
        attn = self.attention(x)
        attn = F.softmax(attn, dim=1)
        
        # Apply attention
        x = x * attn
        
        # Flatten layers
        x = x.view(-1)
        
        # Feed to output layers
        out_emotions = self.emotions(x)
        out_segment = self.segment(x)
        
        return F.sigmoid(out_emotions), F.softmax(out_segment)

Are there any issues with the model? If so, can you suggest the changes and the logic?
Thanks in advance.