BERT will be fine-tuned to perform the classification of my work experience section.
About the maximum number of token per text, I am planning to divide the work experience text in sentences of lenght < 512.
This “raw” split does not allow a perfect division of my job description respect to my date/title/name, so in the description embeddings it could happen that a small amount of context information of date/title/name will be there.
I can easily access to both the “pooled output” (or the hidden layers output and get an average of the last fours layer) and the “token output” of my BERT:
class EntityModel(nn.Module):
   def __init__(self, num_tag):
        super(EntityModel, self).__init__()
        self.num_tag = num_tag
        self.bert = transformers.BertModel.from_pretrained(BASE_MODEL_PATH)
        self.bert_drop_1 = nn.Dropout(0.3)
        self.out_tag = nn.Linear(768, self.num_tag)
    
    def forward(self, ids, mask, token_type_ids, target_pos, target_tag):

       #Forward function of BERT model which return a token output, pooled output and optionally the hidden states' output
        token_output, pooled_output = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids)

        bo_tag = self.bert_drop_1(token_output)

        tag = self.out_tag(bo_tag)

        loss_tag = loss_fn(tag, target_tag, mask, self.num_tag)

        return tag, loss_tag

My original idea was to use the pooled_output if the sentence is a “job description” sentence, so the model will predict only one label for the single output, while if the sentence is not a “job description” sentence, then it is used the token_output, so the model will predict one label for each token.
But this seems to be not possible because the model can’t know the type of the sentence before the prediction.