Can SVM overfit even with cross-validation? I am using SVM regressor models to fit some chemical data related to spectroscopy (I cannot say exactly what data because it is an ongoing research in my group). To combat overfitting, I have used 5-fold cross validation to optimise the hyperparameters with SVM (and other models). The best result is given by SVM.
However, if you look at the plot of SVM predictions vs true values, it looks suspiciously like a case of overfitting.

The hyperparameters that I got after optimizing the average 5-fold cross-validation error are C=1000 and gamma=0.0001 for SVM (I am using scikit-learn). From what I can understand, high value of C and low value of gamma means SVM is going to learn every minute detail from the dataset.
I also had an additional holdout test set (apart from the cross-validation) where I got slightly higher error.

I have two related questions:

*

*Does this look like a case of overfitting? Are there additional tests I can do to check?

*Can overfitting happen even with using cross-validation for hyperparameter optimisation?


Yes, it certainly is possible because you can over-fit the cross-validation statistic when optimising the hyper-parameters.  See
GC Cawley, NLC Talbot, "On over-fitting in model selection and subsequent selection bias in performance evaluation", The Journal of Machine Learning Research 11, 2079-2107 (pdf)
[Full disclosure, this paper was written by Mrs Marsupial and I]
For different samples of data, the cross-validation error as a function of the hyper-parameters will be different, due to random variations in the sampling, and have different minima:

Which can give rise to models that either under- or over-fit the training data (or give good fits):

The best solution is to have more data.  That makes the variance of the cross-validation estimator lower and improves estimation of the hyper-parameters.
This example uses the Least-Squares Support Vector Machine, for convenience, but the standard SVM does pretty much the same thing.  The same thing also happens in a regression setting.  I haven't tried it for SVM regression (because I don't like it very much), but it has been demonstrated for Gaussian Processes, which are likely to be a better option if the dataset isn't too large.
Rekar O Mohammed, Gavin C Cawley, "Over-fitting in model selection with Gaussian process regression", International Conference on Machine Learning and Data Mining in Pattern Recognition, Pages
192-205, 2015 (pdf)
