Which metric to use to evaluate Quantile Regression? I have a prediction problem for which I want to predict the 75% Quantile using Quantile Regression. I am a little bit confused on how to evaluate this model (and also compare different models).
If I understand this approach correctly, for given prediction y_hat the following is true:

*

*With a chance of 75%, the value of y_hat is lower than the actual value of y

*With a chance of 25%, the value of y_hat is higher than the actual value of y
In order to evaluate this model, intuitively I would would compare the overestimations in the test to the underestimations. An "ideal" model would "underestimate" in 25% of the cases (i.e. the value of y_hat is higher than the actual value of y).
Is this a valid approach?
Because I find it somehow confusing to not evaluate regression metrics like MAE, RMSE, Rsquared, etc. But these metrics do no seem suitable here (because we deliberately overestimate). Is there some asymmetric evaluation metric I am missing for this case? Happy if anyone can help clarify my confusion.
Note: I am aware of this post and the respective papers linked there. But is there a corresponding (python) implementation?

@Dave's suggestion is known as the pinball loss, and it is precisely the standard loss function for quantile predictions.
For references, take a look at Koenker's textbook Quantile Regression, or Gneiting (2011, "Quantiles as optimal point forecasts", IJF). We also have a number of threads here at CV.
There are Python implementations in scikit-learn and TensorFlow.
