Overfitting happens for model that is already regularized I am now working on a binary text classification task. The model I am using is as follows

*

*Feature: universal sentence encoder (USE). USE is a feature extractor that is widely used in obtaining representation for longer text like sentence (rather than words).

*Model: Logistic regression (with regularization hyperparameter $C$ optimized on the validation set).

However, I observe the following performance metric on train/val/test set.
SPLIT=train
              precision    recall  f1-score   support

           0       0.79      0.83      0.81      5215
           1       0.73      0.67      0.70      3560

    accuracy                           0.77      8775
   macro avg       0.76      0.75      0.76      8775
weighted avg       0.77      0.77      0.77      8775

SPLIT=val
              precision    recall  f1-score   support

           0       0.75      0.77      0.76       573
           1       0.68      0.65      0.66       427

    accuracy                           0.72      1000
   macro avg       0.71      0.71      0.71      1000
weighted avg       0.72      0.72      0.72      1000

SPLIT=test
              precision    recall  f1-score   support

           0       0.79      0.50      0.61      1604
           1       0.53      0.81      0.64      1115

    accuracy                           0.63      2719
   macro avg       0.66      0.65      0.63      2719
weighted avg       0.68      0.63      0.62      2719



The machine learning 101 tells me the gap between train/test performance gap is probably due to overfitting. However, as this is a binary classification problem, the performance (even on training set) is far from satisfying. Therefore, my question is two-fold

*

*How do I alleviate overfitting for a model that is already regularized (like mine).

*What I could do to improve the performance, especially considering that the training set performance is not good. I know the go-to answer is to try more sophisticated model but what is the guideline for choosing the model?


*

*As @Dave suggests, train first a model that fits the training data. There is no point to talk about overfitting if the model does not fit the data at all. Weighted f1-score of 77% on binary classification is probably not what you would be happy with.


*Such a model may be a random forest. Give a try to xgboost.


*Think about regularisation when you are already fine on training data.


*Poor test score may be caused by different distribution of the test data, not necessarily by overfitting. It seems that your validation set is similar to train set and there the gap is much smaller.
