`T5ForSequenceClassification`
# 🚀 Feature request

T5 to classify sequences by using only the encoder of T5 and a `ClassificationHead`.

## Motivation

This gives the benefits of fine-tuning a model with no maximum sequence length (useful for long sequence tasks) without having to load the decoder weights into memory/treat it as a generative task.

## Your contribution

I already have working code for this, and saw some requests for it in other forums (slack, torch, huggingface) so if it's a welcome addition I'd be happy to add it to the library.

`T5ForMultipleChoice` would also be very helpful.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>This seems like a useful addition, especially considering the EncT5 [paper](https://arxiv.org/pdf/2110.08426.pdf)<|||||>any update on this? <|||||>Maybe of interest to @NielsRogge <|||||>Token Classification would also be very interesting when I think of evaluations for Big Science project.<|||||>But w.r.t. sequence classification, shouldn't it be similar to the sequence classification that is used for the BART model, as seen here:

https://github.com/huggingface/transformers/blob/7799b6128feb17b63c47ed77a71fa367d26492d2/src/transformers/models/bart/modeling_bart.py#L1415-L1530

:thinking: <|||||>Hi. I have done this at https://github.com/subhalingamd/transformers/commit/82db59d553ca870fed69e619d57286bcbed04337. But the list of uninitialized weights doesn't seem convincing at all. Here is the list for reference (for `t5-small`):

```
Some weights of the model checkpoint at t5-small were not used when initializing T5ForSequenceClassification: ['encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'encoder.block.4.layer.1.layer_norm.weight', 'decoder.block.4.layer.0.layer_norm.weight', 'encoder.block.4.layer.1.DenseReluDense.wo.weight', 'decoder.block.4.layer.1.EncDecAttention.v.weight', 'encoder.block.0.layer.1.layer_norm.weight', 'encoder.block.1.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.2.layer_norm.weight', 'decoder.block.2.layer.0.SelfAttention.o.weight', 'encoder.block.1.layer.1.DenseReluDense.wo.weight', 'decoder.block.2.layer.1.EncDecAttention.k.weight', 'encoder.block.2.layer.1.DenseReluDense.wo.weight', 'decoder.final_layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.q.weight', 'decoder.block.2.layer.0.SelfAttention.k.weight', 'decoder.block.5.layer.0.SelfAttention.o.weight', 'encoder.block.0.layer.1.DenseReluDense.wi.weight', 'encoder.block.4.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.1.EncDecAttention.o.weight', 'encoder.block.1.layer.0.SelfAttention.v.weight', 'encoder.block.1.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.0.layer_norm.weight', 'encoder.block.1.layer.0.layer_norm.weight', 'encoder.block.4.layer.0.layer_norm.weight', 'decoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.1.EncDecAttention.v.weight', 'encoder.block.2.layer.0.layer_norm.weight', 'encoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.1.EncDecAttention.k.weight', 'encoder.block.3.layer.1.DenseReluDense.wo.weight', 'encoder.block.3.layer.1.layer_norm.weight', 'encoder.block.0.layer.0.layer_norm.weight', 'decoder.block.2.layer.1.layer_norm.weight', 'decoder.block.2.layer.2.layer_norm.weight', 'decoder.block.0.layer.1.EncDecAttention.v.weight', 'encoder.final_layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.q.weight', 'encoder.block.5.layer.1.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.v.weight', 'encoder.block.2.layer.0.SelfAttention.k.weight', 'encoder.block.3.layer.0.layer_norm.weight', 'encoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.layer_norm.weight', 'decoder.block.3.layer.1.EncDecAttention.k.weight', 'encoder.block.2.layer.0.SelfAttention.v.weight', 'encoder.block.4.layer.0.SelfAttention.q.weight', 'encoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.4.layer.1.EncDecAttention.q.weight', 'decoder.block.2.layer.1.EncDecAttention.v.weight', 'encoder.block.2.layer.0.SelfAttention.o.weight', 'encoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.5.layer.1.EncDecAttention.k.weight', 'encoder.block.0.layer.0.SelfAttention.v.weight', 'encoder.block.5.layer.1.DenseReluDense.wo.weight', 'decoder.block.1.layer.0.SelfAttention.k.weight', 'encoder.block.0.layer.1.DenseReluDense.wo.weight', 'decoder.block.1.layer.1.layer_norm.weight', 'encoder.block.1.layer.0.SelfAttention.q.weight', 'encoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.5.layer.0.SelfAttention.k.weight', 'encoder.block.1.layer.1.layer_norm.weight', 'encoder.block.0.layer.0.SelfAttention.k.weight', 'encoder.block.2.layer.1.DenseReluDense.wi.weight', 'decoder.block.2.layer.2.DenseReluDense.wo.weight', 'encoder.block.5.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.3.layer.1.EncDecAttention.q.weight', 'encoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.2.layer.1.EncDecAttention.o.weight', 'decoder.block.0.layer.0.layer_norm.weight', 'decoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight', 'encoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.2.DenseReluDense.wo.weight', 'shared.weight', 'encoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'decoder.block.3.layer.2.DenseReluDense.wo.weight', 'decoder.block.4.layer.1.EncDecAttention.k.weight', 'decoder.block.4.layer.1.EncDecAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.o.weight', 'encoder.block.1.layer.1.DenseReluDense.wi.weight', 'encoder.block.4.layer.1.DenseReluDense.wi.weight', 'decoder.block.5.layer.2.DenseReluDense.wo.weight', 'encoder.block.4.layer.0.SelfAttention.v.weight', 'decoder.block.1.layer.0.SelfAttention.v.weight', 'decoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.4.layer.2.DenseReluDense.wi.weight', 'decoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.0.layer_norm.weight', 'encoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.1.layer_norm.weight', 'decoder.block.3.layer.2.DenseReluDense.wi.weight', 'decoder.block.1.layer.2.layer_norm.weight', 'decoder.block.5.layer.2.layer_norm.weight', 'decoder.block.1.layer.1.EncDecAttention.v.weight', 'encoder.block.5.layer.1.DenseReluDense.wi.weight', 'encoder.block.5.layer.0.SelfAttention.k.weight', 'decoder.block.0.layer.2.DenseReluDense.wi.weight', 'decoder.block.5.layer.1.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.v.weight', 'encoder.block.3.layer.1.DenseReluDense.wi.weight', 'decoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.4.layer.1.layer_norm.weight', 'decoder.block.2.layer.0.SelfAttention.v.weight', 'decoder.block.4.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.1.EncDecAttention.o.weight', 'decoder.block.5.layer.2.DenseReluDense.wi.weight', 'decoder.block.0.layer.0.SelfAttention.v.weight', 'decoder.block.2.layer.2.DenseReluDense.wi.weight', 'decoder.block.1.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.0.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.1.EncDecAttention.k.weight', 'encoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.1.EncDecAttention.q.weight', 'encoder.block.5.layer.0.layer_norm.weight', 'decoder.block.0.layer.1.EncDecAttention.q.weight', 'decoder.block.1.layer.0.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.k.weight', 'encoder.block.2.layer.1.layer_norm.weight', 'decoder.block.0.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.2.layer_norm.weight', 'decoder.block.1.layer.2.DenseReluDense.wi.weight', 'decoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.4.layer.2.layer_norm.weight', 'decoder.block.2.layer.0.layer_norm.weight']
- This IS expected if you are initializing T5ForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing T5ForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of T5ForSequenceClassification were not initialized from the model checkpoint at t5-small and are newly initialized: ['model.encoder.block.5.layer.0.SelfAttention.k.weight', 'model.decoder.block.4.layer.1.EncDecAttention.k.weight', 'model.decoder.block.4.layer.0.SelfAttention.k.weight', 'model.decoder.block.4.layer.2.DenseReluDense.wi.weight', 'model.decoder.block.3.layer.1.EncDecAttention.v.weight', 'model.decoder.block.0.layer.1.EncDecAttention.v.weight', 'model.decoder.block.5.layer.0.SelfAttention.v.weight', 'model.encoder.block.4.layer.0.SelfAttention.v.weight', 'model.decoder.block.5.layer.0.SelfAttention.q.weight', 'model.decoder.block.2.layer.1.EncDecAttention.k.weight', 'model.encoder.block.1.layer.0.SelfAttention.q.weight', 'model.encoder.block.2.layer.1.DenseReluDense.wo.weight', 'model.encoder.block.3.layer.0.SelfAttention.q.weight', 'model.decoder.block.0.layer.1.EncDecAttention.k.weight', 'model.decoder.block.3.layer.2.DenseReluDense.wi.weight', 'model.encoder.block.4.layer.0.SelfAttention.k.weight', 'model.encoder.block.4.layer.0.layer_norm.weight', 'model.decoder.block.0.layer.0.layer_norm.weight', 'model.decoder.block.4.layer.1.EncDecAttention.v.weight', 'model.decoder.block.3.layer.1.EncDecAttention.k.weight', 'model.decoder.block.2.layer.0.SelfAttention.v.weight', 'model.encoder.block.0.layer.1.layer_norm.weight', 'model.decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'model.encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'model.decoder.block.1.layer.0.SelfAttention.o.weight', 'model.decoder.block.1.layer.0.layer_norm.weight', 'model.encoder.block.4.layer.1.DenseReluDense.wo.weight', 'model.decoder.block.0.layer.0.SelfAttention.k.weight', 'model.decoder.block.0.layer.2.DenseReluDense.wo.weight', 'model.decoder.block.5.layer.1.EncDecAttention.o.weight', 'model.decoder.block.2.layer.0.SelfAttention.o.weight', 'model.decoder.block.3.layer.0.SelfAttention.v.weight', 'model.decoder.block.0.layer.1.EncDecAttention.o.weight', 'classification_head.out_proj.weight', 'model.decoder.block.2.layer.0.SelfAttention.q.weight', 'model.decoder.block.4.layer.0.SelfAttention.v.weight', 'model.encoder.block.3.layer.0.SelfAttention.v.weight', 'model.encoder.block.5.layer.0.SelfAttention.o.weight', 'model.decoder.block.2.layer.0.SelfAttention.k.weight', 'model.decoder.block.3.layer.1.EncDecAttention.o.weight', 'model.encoder.block.2.layer.0.SelfAttention.v.weight', 'model.decoder.block.1.layer.2.DenseReluDense.wi.weight', 'model.decoder.block.4.layer.1.layer_norm.weight', 'model.decoder.block.1.layer.1.EncDecAttention.q.weight', 'model.decoder.block.5.layer.1.EncDecAttention.q.weight', 'model.decoder.block.5.layer.2.DenseReluDense.wi.weight', 'model.encoder.embed_tokens.weight', 'model.encoder.block.1.layer.1.DenseReluDense.wo.weight', 'model.encoder.block.1.layer.1.DenseReluDense.wi.weight', 'model.encoder.block.3.layer.0.layer_norm.weight', 'model.encoder.block.0.layer.1.DenseReluDense.wo.weight', 'model.decoder.block.3.layer.2.layer_norm.weight', 'model.encoder.block.5.layer.1.layer_norm.weight', 'model.encoder.block.3.layer.1.DenseReluDense.wo.weight', 'model.encoder.block.4.layer.1.layer_norm.weight', 'model.decoder.block.3.layer.0.layer_norm.weight', 'model.encoder.block.0.layer.0.SelfAttention.k.weight', 'model.decoder.block.2.layer.0.layer_norm.weight', 'model.decoder.block.0.layer.1.EncDecAttention.q.weight', 'model.encoder.block.4.layer.0.SelfAttention.o.weight', 'model.decoder.block.2.layer.2.layer_norm.weight', 'model.decoder.block.2.layer.2.DenseReluDense.wo.weight', 'model.decoder.block.0.layer.0.SelfAttention.q.weight', 'model.decoder.block.0.layer.0.SelfAttention.v.weight', 'model.encoder.final_layer_norm.weight', 'model.encoder.block.0.layer.0.layer_norm.weight', 'model.encoder.block.3.layer.0.SelfAttention.k.weight', 'model.encoder.block.0.layer.0.SelfAttention.q.weight', 'model.encoder.block.2.layer.1.DenseReluDense.wi.weight', 'model.decoder.final_layer_norm.weight', 'model.decoder.block.4.layer.2.DenseReluDense.wo.weight', 'model.decoder.block.3.layer.0.SelfAttention.q.weight', 'model.encoder.block.2.layer.0.SelfAttention.o.weight', 'model.decoder.block.3.layer.0.SelfAttention.k.weight', 'model.encoder.block.0.layer.0.SelfAttention.o.weight', 'model.encoder.block.4.layer.1.DenseReluDense.wi.weight', 'model.decoder.block.0.layer.2.layer_norm.weight', 'model.decoder.block.1.layer.1.layer_norm.weight', 'model.encoder.block.5.layer.1.DenseReluDense.wo.weight', 'model.encoder.block.1.layer.0.SelfAttention.v.weight', 'model.decoder.block.1.layer.0.SelfAttention.v.weight', 'model.encoder.block.1.layer.1.layer_norm.weight', 'classification_head.dense.bias', 'model.decoder.block.2.layer.1.layer_norm.weight', 'model.decoder.block.3.layer.1.EncDecAttention.q.weight', 'model.decoder.block.5.layer.2.DenseReluDense.wo.weight', 'model.encoder.block.3.layer.1.layer_norm.weight', 'model.decoder.block.1.layer.1.EncDecAttention.k.weight', 'model.decoder.block.0.layer.0.SelfAttention.o.weight', 'model.encoder.block.2.layer.0.SelfAttention.q.weight', 'model.decoder.block.5.layer.0.SelfAttention.o.weight', 'model.decoder.block.4.layer.0.SelfAttention.o.weight', 'model.decoder.embed_tokens.weight', 'model.decoder.block.2.layer.2.DenseReluDense.wi.weight', 'model.encoder.block.3.layer.0.SelfAttention.o.weight', 'model.encoder.block.0.layer.1.DenseReluDense.wi.weight', 'model.encoder.block.1.layer.0.SelfAttention.o.weight', 'model.decoder.block.0.layer.1.layer_norm.weight', 'model.decoder.block.3.layer.0.SelfAttention.o.weight', 'classification_head.dense.weight', 'model.encoder.block.5.layer.0.SelfAttention.v.weight', 'model.decoder.block.5.layer.1.EncDecAttention.v.weight', 'model.decoder.block.3.layer.2.DenseReluDense.wo.weight', 'model.decoder.block.4.layer.1.EncDecAttention.o.weight', 'model.decoder.block.2.layer.1.EncDecAttention.o.weight', 'model.decoder.block.4.layer.1.EncDecAttention.q.weight', 'model.shared.weight', 'model.decoder.block.4.layer.0.layer_norm.weight', 'model.encoder.block.5.layer.0.layer_norm.weight', 'model.encoder.block.5.layer.1.DenseReluDense.wi.weight', 'model.decoder.block.2.layer.1.EncDecAttention.q.weight', 'model.decoder.block.3.layer.1.layer_norm.weight', 'model.decoder.block.5.layer.1.layer_norm.weight', 'model.encoder.block.2.layer.1.layer_norm.weight', 'model.encoder.block.0.layer.0.SelfAttention.v.weight', 'model.encoder.block.2.layer.0.SelfAttention.k.weight', 'model.decoder.block.4.layer.0.SelfAttention.q.weight', 'model.encoder.block.1.layer.0.layer_norm.weight', 'model.decoder.block.1.layer.1.EncDecAttention.v.weight', 'model.decoder.block.0.layer.2.DenseReluDense.wi.weight', 'model.encoder.block.4.layer.0.SelfAttention.q.weight', 'model.decoder.block.5.layer.2.layer_norm.weight', 'model.decoder.block.5.layer.1.EncDecAttention.k.weight', 'model.encoder.block.3.layer.1.DenseReluDense.wi.weight', 'classification_head.out_proj.bias', 'model.encoder.block.5.layer.0.SelfAttention.q.weight', 'model.decoder.block.1.layer.0.SelfAttention.q.weight', 'model.decoder.block.1.layer.1.EncDecAttention.o.weight', 'model.decoder.block.1.layer.2.DenseReluDense.wo.weight', 'model.encoder.block.2.layer.0.layer_norm.weight', 'model.decoder.block.1.layer.0.SelfAttention.k.weight', 'model.decoder.block.5.layer.0.layer_norm.weight', 'model.encoder.block.1.layer.0.SelfAttention.k.weight', 'model.decoder.block.4.layer.2.layer_norm.weight', 'model.decoder.block.1.layer.2.layer_norm.weight', 'model.decoder.block.5.layer.0.SelfAttention.k.weight', 'model.decoder.block.2.layer.1.EncDecAttention.v.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```


In case of `BartForSequenceClassification` with `facebook/bart-large`, this is how it looks like:
```
Some weights of BartForSequenceClassification were not initialized from the model checkpoint at facebook/bart-large and are newly initialized: ['classification_head.dense.weight', 'classification_head.dense.bias', 'classification_head.out_proj.weight', 'classification_head.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```



Am I missing out on something, or is it just that something extra has to be done to import the model? 🤔<|||||>@stefan-it / @subhalingamd  the code for `BartForSequenceClassification` loads both the encoder and decoder parts for BART, which doesn't follow from the EncT5 paper - `model` should be `T5Encoder` only.

My solution is [here](https://github.com/MetcalfeTom/transformers/pull/1/files), happy to push it, though it's a lot of duplicate code.  Should some refactoring be performed between this and Bart?

The other addition from the EncT5 paper is that the encoder outputs are pooled to simulate the text-to-text nature of classification using NLG.  This is not in my implementation but can be added.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@LysandreJik 
can we expect this to be added sometime soon?<|||||>Hi,
I don't know if anyone is still working on this, but looking to recent works that uses just the encoder from T5 I implemented the class `T5ForSequenceClassification` and evaluated on GLUE.

Here is the repository with the code and results: https://github.com/osainz59/t5-encoder 

Is there still an interest on implementing this into Transformers library?<|||||>Hi @osainz59 , your repo looks really interesting, did you also perform experiments on NER? I've recently added support for encoder-only fine-tuning into Flair library, and results are really promising for this kind of downstream task. However, both sequence and token classification would be awesome to have it in Transformers :hugs: <|||||>Hi @stefan-it , I did not test the `T5ForTokenClassification` yet since there is no direct comparison afaik to traditional T5. However I can run it on some datasets to ensure it works properly. Can you tell me which datasets might be of interest to test?<|||||>Hi @osainz59 I think one really interesting dataset would be the CoNLL-2003 (see https://huggingface.co/datasets/conll2003). 

When testing the mT5 model series, the WikiANN (Rahimi splits from here: https://huggingface.co/datasets/wikiann) is also very interesting (train on English split only and test it on the other languages for comparisons with the mT5 paper) :)<|||||>Hi @stefan-it , I trained and evaluated the `T5ForTokenClassification` class on CoNLL-2003 and here are the results:
```
***** eval metrics *****
  epoch                   =       25.0
  eval_accuracy           =     0.9916
  eval_f1                 =     0.9549
  eval_loss               =     0.0449
  eval_precision          =     0.9531
  eval_recall             =     0.9567
  eval_runtime            = 0:00:10.90
  eval_samples            =       3251
  eval_samples_per_second =    298.079
  eval_steps_per_second   =     37.317
```

I think they are still a bit behind of RoBERTa, but at those levels of F1 is hard to decide. Nevertheless, I think these results suggests that T5-Enc could be an interesting addition to the Transformers library. <|||||>Hey @osainz59 thanks for reporting back! I would love to see this in Transformers directly!<|||||>> Hi, I don't know if anyone is still working on this, but looking to recent works that uses just the encoder from T5 I implemented the class `T5ForSequenceClassification` and evaluated on GLUE.
> 
> Here is the repository with the code and results: https://github.com/osainz59/t5-encoder
> 
> Is there still an interest on implementing this into Transformers library?

This is exactly what I am looking for!<|||||>I've opened a PR https://github.com/huggingface/transformers/pull/24726 for `T5ForSequenceClassification` following the structure of the `BartForSequenceClassification` so both Encoder and Decoder weights are being used. 

Although based on the results shown in this thread it seems like we could also look into adding a version that only uses the Encoder as well. 