Calculate intra-class correlation (or some other measure of reliability) with variable number of raters I have some stimuli rated by users. I want to estimate how much of the variance in ratings is due to the stimuli, so that I can compare between various types of ratings which are the most reliable. This seems like it is an Intra-Class Correlation (ICC(1,1): one-way random effects), but my problem is that packages to do this in R (psych) or python (pingouin) all assume that the ratings come from the same set of raters, while in my data every single rating was made by a different individual and the number of ratings per stimuli varies between 3 and 5,000.
My dataset looks like:
STIMULI       RATINGS_FEATURE1
A             [1,3,2,1,3]
B             [4,3]
C             [6,7,7,7,7,7,7,7,7,7,7,7,5,4,1,8,6,7,7,7,7,7,7,7,7,7,7]
D             [3,2,3,3,1]
....

What do I want to do?

Correlation is a statistic that relates to paired measurements. Intraclass correlation is a generalisation that extends the pairs to groups.

while in my data every single rating was made by a different individual

If you don't have grouped data, then correlation makes no sense.

I want to estimate how much of the variance in ratings is due to the stimuli,

This sounds more like ANOVA or a Kruskal–Wallis test.
