How to add traditional features for transformers?
## ❓ Questions & Help

I don't know whether models like Bert can capture some traditional features or not, such as the number of nouns in a sentence, the number of words in a sentence, etc. But I want to add these features to Bert or xlnet for training. Can anyone help me?

What do you mean "for training"? Training what? I imagine that you wish to do this for a downstream task. In that case you can just use BERT for feature extraction and use its features in another system. I suppose one could also fine-tune the model during training where you somewhere inject other features into the model, but you'll have to think carefully where and how you optimize the model (one optimizer for finetuning and training newly injected parameters, multiple optimizers, and so on).

I'd say you could start with feature extraction and adding other features. <|||||>As correctly said by @BramVanroy , you can see some examples of this approach [here](https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/).

> ##  Questions & Help
> I don't know whether models like Bert can capture some traditional features or not, such as the number of nouns in a sentence, the number of words in a sentence, etc. But I want to add these features to Bert or xlnet for training. Can anyone help me?<|||||>Hi @TheEdoardo93 , I read the approach you told me. But I still have some confusing, since the model could output a vector of 768 for every sentence. Where I concat my feature like "the number of nouns in a sentence"? Just the end line of the vector and normalized all of them to 1? Thank you.<|||||>`BERT` gives you a feature vector of size 768 for each sentence. Besides this, you can add _N_ features to this vector in order to have at the end a vector of size 768 + _N_, in which the N features have been chosen by you, e.g. number of nouns in a sentence.
So, from 0 to 767 you have the features extracted with `BERT` model, and from 768 to 768+_N_-1 you have your **custom** features extracted by you.

N.B: remember that it's very important in Machine Learning to have the range for all features the same (or quite the same), in order to not give more importance to some rather than others. For this task, you can use e.g. StandardScaler or MinMaxScaler from [Scikit-learn](https://scikit-learn.org/stable/) or a custom scaler implemented by you. 


> Hi @TheEdoardo93 , I read the approach you told me. But I still have some confusing, since the model could output a vector of 768 for every sentence. Where I concat my feature like "the number of nouns in a sentence"? Just the end line of the vector and normalized all of them to 1? Thank you.<|||||>@TheEdoardo93 Thank you very much !