What is the best way to validate whether a set of given scores are a good predictor of a success metric? For example, I have a table of items each with the given scores (no unit) and success metric (quantity value). The success metric is the real-life performance indicating number of items sold, etc. whereas the given scores were the predicted success (unitless metric).

*

*What is a good data science method for validating whether these scores do a good job indicating the success metric? I can create a prediction model with mediocre accuracy.

*Are there other methods that can be used specifically to validate as in this case?





Item ID
Score
Success Metric




1
60
982


2
88
1000


3
46
70




The simplest approach would indeed be to use your Score to model your Success Metric in appropriate ways. For instance, you could use OLS if you are looking for unbiased predictions of the Success Metric, or quantile regression if you are looking for quantiles (common in sales prediction, to automatically include safety stocks). You could use spline transforms for your Score if there is nonlinearity. Or use a nonlinear method with an appropriate error metric (Kolassa, 2020).
Of course, this kind of begs the question of why you are modeling a Score to predict the Success Metric, and not modeling the Success Metric directly in the first place.
What is good will depend on your context. A sales forecast with an MSE of 100 may be "good enough", or it may not be worth the effort of setting up the model. A good way to look at this is to compare your model to a simple naive benchmark model, like the overall historical mean or quantile. If you can't beat that, then you indeed have to rethink what you are doing.
