What is a good metric for my item ordering/ranking task? My dataset is a collection of items with numerical features, and each of them has a score assigned to it as label.
My goal is to predict the ranking/order of the items, so the score prediction accuracy is less of concern, as long as the ranking by predicted socres are correct.
My question is, is there a good metric to evaluate my model on this task? I have checked out some learning to rank metrics, but they tend to emphasize the correct order of top items. However, in my case, it is equally important to put trailing items where they belong.
For example,
rank_metric([A, B, C, D, E], [A, B, C, D, E]) should be perfect
rank_metric([A, B, C, D, E], [B, A, C, D, E]) should be slightly lower
rank_metric([A, B, C, D, E], [A, B, D, C, E]) should be the same as the previous one
rank_metric([A, B, C, D, E], [D, B, C, A, E]) should be punished even more
Thanks in advance!

Supplementing earlier answer
Keyword:  Learning To Rank ( LTR) problem formulation 
Three prominent methods to solve ranking problems.

*

*Pointwise ranking model

*Pairwise ranking model

*Listwise ranking model
See https://en.wikipedia.org/wiki/Learning_to_rank & links therein. A very rich & matured field of machine learning.

