Why would a Linear SVM perform worse than Logistic Regression? I have a dataset of about 1000 features and 2000 training examples. I use Randomized Search with cross validation to compare Random Forest, Linear SVC, SVC with different non-linear kernels, and Logistic Regression.
I consistently get much worse scores with Linear SVC than with the other models, including Logistic regression (AUC ROC of 0.70 for Linear SVC vs 0.80 for other models).
From what I understand, the performance of the Linear SVMs should be comparable to Logistic Regression. What could be the reason for such poor performance?
Here are the hyperparamaters I'm checking:
rf_param_grid = {
    'randomforestclassifier__max_depth' : np.random.randint(5, 150, 30),
    'randomforestclassifier__min_samples_split': np.random.randint(2, 50, 30),
    'randomforestclassifier__n_estimators': np.random.randint(50, 400, 10),
    'randomforestclassifier__min_samples_leaf': np.random.randint(1, 20, 30),
    'randomforestclassifier__max_features': ['auto', 'sqrt', 'log2', 0.25, 0.5, 0.75, 1.0],
    'randomforestclassifier__criterion': ['gini', 'entropy'],
    'randomforestclassifier__class_weight':["balanced", "balanced_subsample", None],
    "randomforestclassifier__class_weight": ['balanced', None]
}
linear_svc_param_grid = {
    'svc__C': [0.1, 0.2, 0.3, 0.4, 0.5, 1, 5, 10],
    "svc__class_weight": ['balanced', None]
}
kernel_svc_param_grid = {
    'svc__C': loguniform(1e-1, 1e3),
    'svc__gamma': loguniform(1e-04, 1e+01),
    'svc__degree': uniform(2, 5),
    'svc__kernel': ['poly', 'rbf', 'sigmoid'],
    "svc__class_weight": ['balanced', None]
}
lr_param_grid = {
    'logisticregression__C': loguniform(1e-5, 1e4),
    'logisticregression__penalty': ['l1', 'l2', 'elasticnet'],
    'logisticregression__class_weight': ['balanced', None],
    'logisticregression__l1_ratio': uniform(0, 1)
}


Maybe the linear SVC grid doesn't cover enough of the parameter space, or the grid is too coarse. You only specify 8 grid values for linear SVC. You don't tell us anything about the random search configuration, but if you're using more than 8 random search iterations, you're just repeatedly testing one or more of those same 8 values for SVC. There might be a hyper-parameter value, not among those 8, that improves the model to be consistent with your expectations.
For other hyper-parameters, you use continuous probability distributions. That means that each random search iteration draws a random value that is different than every value you've already tested. Using the same approach for linear SVC lets you test new hyper-parameter values at each random search iteration.
