Fine tuning with transformer models for Regression tasks
- `transformers` version: Bert, Albert, openai-gpt2
- Tensorflow version (GPU?): 2.5.0


## Information

Model I am using :  Bert, Albert, openai-gpt2

The problem arises when using:
* [x] my own modified scripts: (give details below) <br>
- performed fine tuning


The tasks I am working on is:

* [x] my own task or dataset: (give details below)<br>
- I have been trying to use BertModel, albert and GPT2 models for fine tuning on my regression task and i was able to produce unwanted results . i will mention it below: <br>

- I tried it two times: <br>

1. I used CLS token embeddings and fine tuned over my entire custom model but it produced some random number repeating over and over in my output matrix space.<br>

2. I simply passed CLS token embeddings to the feed forward NN. In this case also it produced some random number and no learning is seen here.<br>

<br>

**what can be the solution to this problem? is there any issues with transformers with respect to regression?**
I think it's better to ask this question on the [forum](https://discuss.huggingface.co/) rather than here. For example, all questions related to training BERT for regression can be found [here](https://discuss.huggingface.co/search?q=bert%20regression).