It depends on the model. BERT uses the first one (where the CLS token is), some models use a pooling of all hidden states, other the one for the last logits (which is not necessarily -1 since you could have padding). I’d look at what is done in T5ForSequenceClassification and copy the code.