Overfitting happens for model that is already regularized I am now working on a binary text classification task. The model I am using is as follows

*

*Feature: universal sentence encoder (USE). USE is a feature extractor that is widely used in obtaining representation for longer text like sentence (rather than words).

*Model: Logistic regression (with regularization hyperparameter $C$ optimized on the validation set).

However, I observe the following performance metric on train/val/test set.
SPLIT=train
              precision    recall  f1-score   support

           0       0.79      0.83      0.81      5215
           1       0.73      0.67      0.70      3560

    accuracy                           0.77      8775
   macro avg       0.76      0.75      0.76      8775
weighted avg       0.77      0.77      0.77      8775

SPLIT=val
              precision    recall  f1-score   support

           0       0.75      0.77      0.76       573
           1       0.68      0.65      0.66       427

    accuracy                           0.72      1000
   macro avg       0.71      0.71      0.71      1000
weighted avg       0.72      0.72      0.72      1000

SPLIT=test
              precision    recall  f1-score   support

           0       0.79      0.50      0.61      1604
           1       0.53      0.81      0.64      1115

    accuracy                           0.63      2719
   macro avg       0.66      0.65      0.63      2719
weighted avg       0.68      0.63      0.62      2719



The machine learning 101 tells me the gap between train/test performance gap is probably due to overfitting. However, as this is a binary classification problem, the performance (even on training set) is far from satisfying. Therefore, my question is two-fold

*

*How do I alleviate overfitting for a model that is already regularized (like mine).

*What I could do to improve the performance, especially considering that the training set performance is not good. I know the go-to answer is to try more sophisticated model but what is the guideline for choosing the model?


Your dataset is far too small for split sample validation to be reliable.  Use a resampling method such as the bootstrap or 100 repeats of 10-fold cross-validation.  You are using logistic regression, which is a probability model, but then you are discarding the probabilities and reverting to a classification system, and using discontinuous accuracy scores that will be fooled by a bogus model.  For more see this.  For better modeling and validation strategies related to probability models see RMS.
