How damaging to the analysis would it be to run probability validation (`rms::val.prob`) when calibration (`rms::calibrate`) is the correct action? If I make a model that predicts probabilities (e.g., logistic regression or a neural network), I would like it to have the property that, when it predicts a probability of $p$, the event happens about $100p\%$ of the time. That is, I would like calibrated probabilities.
If I do my work in Frank Harrell’s rms R package, then I can just stick my probability predictor into the calibrate function and get a nice graph that illustrates how truthful the predicted probability values are.
However, if I do not create my model in that package, Harrell’s function need not apply to the model object from another package, and so I cannot run the calibration function on my model. Further, even if I write my own version of the function that will apply to the model object I have created, if I have a gigantic model that takes hours or days to train, the bootstrap procedure in rms::calibrate will take longer than I might be able to tolerate.
There is a related function, rms::val.prob, that checks if claimed probabilities are aligned with observed outcomes. Unlike calibration, probability validation does not consider the model-building procedure, so if I created my probabilities using a neural network that takes days to train, all I have to do is input the predictions and observed outcomes, rather than training a new model many times. This could be a substantial saver of computing time, and even if the computing time would be quick, I do not have to write (and test…) my own software to operate on a regression object from outside of rms.
The “correct” approach is to use calibration in this situation: the predicted probabilities come from a model. However, what is lost by applying the quicker probability validation?
Related Questions of Mine
Walk through calibration
Walk through probability validation
EDIT
As is discussed in the comments, I think the crux here is the optimism bootstrap in the calibration function and what we lose when we apply probability validation instead of retraining models (as is performed in the calibration function).

The manual page for the val.prob() function in the rms package says:

Given a set of predicted probabilities p or predicted log odds logit, and a vector of binary outcomes y that were not used in developing the predictions p or logit, val.prob computes the following indexes and statistics:...(Emphasis added).

So the choice might be better phrased in terms of internal versus external validation, as Harrell discusses here and in Section 5.3 of Regression Modeling Strategies. The val.prob() function would be OK for external validation but not for internal validation. Any proper internal validation method will be based on resampling. If resampling is too costly, then the tradeoffs are those involved with train/test data splits.
