Compare the bootstrap results of 2 datasets with split-half reliability I am currently working on the analysis of a metric comparing two methods (A & B) for subjective annotation. The two methods give 2 types of data which after processing gives scores relatives to the subjective dimension studied for a dataset of item.
In order to compare their performance, I do a reliability test called Split Half Reliability(SHR) that splits the dataset of each method's results in two subsets and compute a correlation coefficient between them. The highest correlation coefficient between two subsets means high reliability. However, there is a lot of variance depending on the organization of the data in each subsets, doing one SHR is therefore not accurate.
So I want to do a bootstrap procedure for each of the method with these steps:
For the two methods'results simultaneously :
1/ sample with replacement the participants data
2/ compute scores for the whole dataset
3/ compute SHR (correlation coefficient, for each method's dataset)
4/ redo steps 1-3 N times
At the end, I'm thinking of a T-test if the distributions of the two methods are normal.
I'm just not sure I'm allowed to follow these steps...
I would gladly take advice (and references if possible).
Thank you :)

I don't think the split half reliability is the best approach here. The problem with SHR is, as you say, that you can get a different value according to how you split.
Coefficient (Cronbach's) alpha is the average of all the possible split half correlations (it's also a form of ICC). You can calculate the standard error and do a t-test, or you can use a bootstrap, to compare them.
