I need to make a feature extractor for a project, so I am able to translate a given financial statement (text) into a vector that can be used as features in my main problem. I am currently doing revenue forecasting. I use historical fundamentals data in addition to stock prices in order to predict revenue growth for next quarter (regression problem). In addition I use text data (financial statements) where I want to use BERT in order to get new features for my regression model. That is, the vector from the BERT feature extraction will later be combined with several other values (fundamentals and stock price data) for the final prediction (next quarter revenue growth) in e.g. a random forest or XGBoost model.
I want to try both a FINE-TUNED FinBERT model and a PRE-TRAINED FinBERT MODEL and compare. But how do I fine-tune the FinBERT model on my dataset (regression problem) and then use that new FinBERT model to do the feature extraction?