Metrics We use several metrics to compare how
the PLMs align with group preferences across lan-
guages. These include top-k precision P@k with
k={1, 5}, mean reciprocal rank (MRR), and two
classical univariate rank correlations: Spearman’s ρ
(Spearman, 1987) and Kendall’s τ (Kendall, 1938).
Given a set of |S| cloze sentences and a group
of annotators, for each sentence s, we denote
the list of answers, ranked by their frequency, as
Ws = [w1, w2, ...], and the list of model’s predic-
tions as Cs = [c1, c2, ...], ranked by their model
likelihood. Then, we report P@k = 1[ci ∈ Ws]
with i ∈ [1, k], where 1[·] is the indicator function.
Precision is reported together with its standard de-
viation, to account for the group-wise disparity in

7We use the base models available from https://
huggingface.co/models. We report results using un-
cased mBERT, since it performed better on our data than its
cased sibling.

3598(cid:115)(cid:80)G

both dimensions (social groups and language):

σgd =
Are Pretrained Multilingual Models Equally Fair Across Languages?