You should get a warning, which points to a shape mismatch:

UserWarning: Using a target size (torch.Size([32])) that is different to the input size (torch.Size([32, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
return F.mse_loss(input, target, reduction=self.reduction)

Use loss = loss_function(tag_scores, l.unsqueeze(1)) and your model should train better.