Calculate intra-class correlation (or some other measure of reliability) with variable number of raters I have some stimuli rated by users. I want to estimate how much of the variance in ratings is due to the stimuli, so that I can compare between various types of ratings which are the most reliable. This seems like it is an Intra-Class Correlation (ICC(1,1): one-way random effects), but my problem is that packages to do this in R (psych) or python (pingouin) all assume that the ratings come from the same set of raters, while in my data every single rating was made by a different individual and the number of ratings per stimuli varies between 3 and 5,000.
My dataset looks like:
STIMULI       RATINGS_FEATURE1
A             [1,3,2,1,3]
B             [4,3]
C             [6,7,7,7,7,7,7,7,7,7,7,7,5,4,1,8,6,7,7,7,7,7,7,7,7,7,7]
D             [3,2,3,3,1]
....

What do I want to do?

The way you've collected your data precludes any calculation of 'reliability'. The first part of your question asks:

...how much of the variance in ratings is due to the stimuli, ...

You can answer this with a standard mixed effect model i.e.,
Ratings ~ 1 + (1|Stimuli)
Using this framework you could estimate the total repeatability across all stimuli. See for example the rptR package.
Edit
To expand on my answer, take example data from the rptR package. Your data is unbalanced, so we'll create a new unbalanced data set from the example data.
library(rptR)
library(lme4)
data(BeetlesBody)

set.seed(12345)
BeetlesBody_unbalanced = BeetlesBody[sample(c(1:960),1000,replace = T),]
table(BeetlesBody_unbalanced $Population)

  1   2   3   4   5   6   7   8   9  10  11  12 
106  80  87  83  73  71  65  92  93  79  87  84 

We can compute the repeatability as follows:
rpt(BodyL ~  (1 | Population), grname = "Population", data = BeetlesBody, datatype = "Gaussian", 
+     nboot = 0, npermut = 0)


Repeatability estimation using the lmm method 

Repeatability for Population
R  = 0.299
SE =  NA 
CI = [NA, NA]
P  = 8.08e-62 [LRT]
     NA [Permutation]

Our repeatability is 0.299. Note that repeatability is just a ratio composed of the random effects variances. See the rptR package vignette for more on this. Things get more complicated if your model includes fixed effects and you want to account for the variance explained in your calculation of repeatability.
summary(lmer(BodyL ~ 1 + (1 | Population),data = BeetlesBody))

Linear mixed model fit by REML ['lmerMod']
Formula: BodyL ~ 1 + (1 | Population)
   Data: BeetlesBody

REML criterion at convergence: 3893.3

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.2950 -0.7770  0.0186  0.7867  2.6107 

Random effects:
 Groups     Name        Variance Std.Dev.
 Population (Intercept) 1.377    1.173   
 Residual               3.235    1.798   
Number of obs: 960, groups:  Population, 12

Fixed effects:
            Estimate Std. Error t value
(Intercept)  14.0827     0.3436   40.98

That is, repeatability = 1.377/(1.377+3.235) = 0.299
