Cox prediction models: Statistical inference versus cohort-split (derivation->test) First I want to clarify that I understand that all prediction models needs external validation, and this applies to both machine learning models and conventional regression models. My question is regarding the development of prediction model using Cox-regression and the need for internal validation.
Machine learning models, which iteratively adjusts the weighting of prediction-variables trough a loss function, are prone to overfitting, and it is easy to understand why both internal and external validation is necessary. Perhaps the one could use some alternatives to the Train-test-splitting with boot-strap-aggregating methods or multi-fold cross-validation, but this is another issue.
The question is: In regression methods where you fit the model to the data without iterative adapting the variable coefficients: Does this approach really need the same form of internal validation by splitting in a derivation cohort and a test cohort? If so, what does this really achieve?
I understand some situations where this intuitively can be useful: 1)If you have a temporal difference in sub-cohorts where you compare older cases with newer cases 2) Other identifiable differences in the cohort that you can categorize the data by(for example variables potentially representing a source of noise). However, it is not clear to me why one should expect different results when random splitting a sample into a derivation sample and a test sample, fitting a model on the derivation sample and testing on the test-sample, compared to fitting the model on the whole sample and evaluating the predictive potential by inference from the sample.
It is of course obligatory to perform external validations to evaluate if the model predictions are valid for external data, before even thinking about to actually use a model in real-life-settings.
I have tried doing random splits in several datasets, and I never get different results when evaluating the model by splitting and predicting on test set, compared to the inferential approach.
If anyone have some enlightening insights on the matter, it would be greatly appreciated.

A table from real life data, illustrating the issue, with a "positive" optimism estimate:

