I have the output of a Bert model that is H = [batch_size, sequence_length, hidden_size]. I have another tensor indicating the sequence length of each sample in the batch, L = [batch_size, 1].
How would I compute the max of H over the sequence length (dim=1) without including the padding tokens (i.e. only including the positions less than that specificied by L)?
I have a solution for taking the mean,
mask = torch.arange(H.shape[1]).repeat(H.shape[0], 1)
mask = (mask < L).float()
mask[:, 0] = 0
masked_h = H * mask.unsqueeze(2)
mean_emb = (masked_h.sum(dim=1)/L)