Figure 1 complements results from Table 2 with
MRR scores. We observe a common trend that
the models often underperform on non-native male
speakers in all languages except for Spanish: Per-
formance is (always) below the average, and they
are the worst-off group (↓) in most of the cases.
At the same time, predictions with mBERT and

13.2 (0.3)

11.8 (1.1)

10.8 (3.5)

7.3 (2.5)

P@1
MN
FN
MNN
FNN

P@1
MN
FN
MNN
FNN

P@1
MN
FN
MNN
FNN

EN
13.3
13.3
12.7
13.3

EN
16.7
16.0
15.3
20.0

EN
2.0
4.0
2.0
3.3

mBERT
DE
11.3
15.3
11.4
5.6

XLM-R

DE
20.7
24.0
15.0
13.1

DE
8.7
6.7
6.4
1.9

mT5

ES
12.7
12.0
12.4
10.0

ES
13.3
15.3
13.5
14.7

ES
4.7
3.3
4.7
6.7

FR
10.7
8.0
3.6
6.9

12.0 (1.0)
12.2 (2.7)
10.0 (3.8)
9.0 (3.0)
P@1(σgd)

FR
16.7
17.3
11.4
12.7

16.9 (2.6)
18.2 (3.5)
13.8 (1.5)
15.1 (3.0)
14.5 (2.6) P@1(σgd)

FR
5.3
3.3
4.3
6.2

5.2 (2.4)
4.3 (1.4)
4.4 (1.6)
4.5 (2.0)
P@1(σgd)

17.0 (1.8)

14.2 (0.8)

18.2 (4.4)

2.8 (0.9)

4.8 (1.2)

5.8 (2.5)

4.8 (1.1)
Are Pretrained Multilingual Models Equally Fair Across Languages?