Let's have a look inside the token classification pipeline. In the pipeline video, we looked at the different applications the Transformers library supports out of the box, one of them being token classification, for instance predicting for each word in a sentence whether they correspond to a person, an organization or a location. We can even group together the tokens corresponding to the same entity, for instance all the tokens that formed the word Sylvain here, or Hugging and Face. The token classification pipeline works the same way as the text classification pipeline we studied in a previous video. There are three steps: the tokenization, the model, and the post processing. The first two steps are identical to the text classification pipeline, except we use an auto token classification model instead of a sequence classification one. We tokenize our text then feed it to the model. Instead of getting one number for each possible label for the whole sentence, we get one number for each of the possible 9 labels for every token in the sentence, here 19. Like all the other models of the Transformers library, our model outputs logits, which we turn into predictions by using a SoftMax. We also get the predicted label for each token by taking the maximum prediction (since the softmax function preserves the order, we could have done it on the logits if we had no need of the predictions). The model config contains the label mapping in its id2label field. Using it, we can map every token to its corresponding label. The label O correspond to "no entity", which is why we didn't see it in our results in the first slide. On top of the label and the probability, those results included the start and end character in the sentence. We will need to use the offset mapping of the tokenizer to get those (look at the video linked below if you don't know about them already). Then, looping through each token that has a label distinct from O, we can build the list of results we got with our first pipeline. The last step is to group together tokens that correspond to the same entity. This is why we had two labels for each type of entity: I-PER and B-PER for instance. It allows us to know if a token is in the same entity as the previous one.() Note that there are two ways of labelling used for token classification, one (in pink here) uses the B-PER label at the beginning of each new entity, but the other (in blue) only uses it to separate two adjacent entities of the same type. In both cases, we can flag a new entity each time we see a new label appearing (either with the I or B prefix) then take all the following tokens labelled the same, with an I-flag. This, coupled with the offset mapping to get the start and end characters allows us to get the span of texts for each entity.