@lewtun I have a question about this. So trying to summarize, If we take a model like distillbert for the purposes of extracting embeddings as features for a downstream task, I have three options:

Use pretrained model out of the box with no fine tuning (as in chapter 2 “Transformers as Feature Extractors”)
Fine-tune the model as a masked language model with the unlabelled corpus to update the vocabulary ( as in Fine-tuning a masked language model)
Fine-tune the model with the labelled corpus, to ensure the embeddings encapsulate the information needed for separating the classes

And from each of these I can extract embeddings, and then use them as features in any classification task I need.
In my particular case I have ~1m unlabelled text descriptions, and about 50K labelled descriptions. Would it make sense to do all three here? I.e. get the pretrained model, expand the domain vocabulary as much as possible (mlm fine-tune), and then tune for classification (SequenceClassification fine tuning)? Would either be more likely to have more impact than the other in this case (mlm fine-tune vs classification fine-tune)? Would the order matter? I realize the answer may be to test it and evaluate but curious if there’s any rules of thumb.

Apply traditional feature engineering. Besides using the embeddings from Transformer models, we could also add features such as the length of the tweet or whether certain emojis or hashtags are present

Are there any resources how to best do this? Sorry if this is covered in the second half of the book! I am finding it very helpful so far   Would the approach be like the sklearn example, where the embeddings are read into a dataframe and I just add additional features to it and then train a classifier? I’m curious how scaling etc is handled when combining other features with embeddings like that.
Also it feels like the sklearn classifier is doing the same job of the final activation layer in a network. Would there be a way to achieve this in a single neural network, instead of training one network, extracting embeddings from it and then using it to train a separate classifier model? So in the example from the book, one could add the tweet length to the embedding vector somehow and still treat it as a regular SequenceClassification problem? To avoid having to optimize them separately/sequentially. Thanks again for all the great resources here