How to check if two lists of rankings lists are statistically significant different To make my problem more clear I will give some examples of my data:





Measure A
Measure B




Set 1
5 3 1 4 2
5 4 2 3 1


Set 2
5 1 3 4 2
5 4 1 3 2


Set 3
2 4 3 5 1
5 2 3 4 1


Set 4
3 4 1 5 2
5 3 2 4 1


Set 5
5 3 2 4 1
5 4 2 3 1




I have multiple sets of items to be ranked and multiple measures by which the items in a set can be ranked. Each cell in the table presents a ranking of the items in one set by one of the measures (e.g. rankings of the 5 items in set 1 as ranked by measure A  are in the cell in at line 2 column 2 of the 3x3 table).
My goal is to check if Measure A ranks items differently than Measure B. I want to check if there is a statistically significant difference between the way that the two measures ranked the items in the different sets.
While searching I only found ways to compare 2 rankings between them (e.g. comparing the ranking of measure A on set 1 and the ranking of measure B on Set 1), but not ways to compare multiple rankings lists.

Your problem might be attacked by performing a permutations test on the ranks for each of the items in the sets (i.e. those labelled '1' through '5' in your table) to determine how strongly the data cast doubt on the hypothesis that the rankings differ between the two measures (methods?) only randomly.
Take each item and tabulate their ranks according to the two methods measures. (Using numbers to represent the items in your table is here not entirely helpful, so maybe use a,b,c... instead.) The ranks for item '5' in the table are 1,1,4,4,1 for measure A and 1,1,1,1,1 for measure B. The sum of the differences in ranks would be a useable statistic here, so a summary if the differences between rankings is 6 (i.e. (1-1)+(1-1)+(4-1)+(4-1)+(1-1)=6). That statistic treats the rankings as paired across measures.
Is 6 large or small? See where it lies in the distribution of all possible re-arrangements of the ranks across the measures by permutations.
