Gender bias is largely recognized as a problematic phenomenon affecting language technologies, with recent studies underscoring that it might surface differently across languages. However, most of current evaluation practices adopt a word-level focus on a narrow set of occupational nouns under synthetic conditions. Such protocols overlook key features of grammatical gender languages, which are characterized by morphosyntactic chains of gender agreement, marked on a variety of lexical items and parts-of-speech (POS). To overcome this limitation, we enrich the natural, gender-sensitive MuST-SHE corpus (Ben-tivogli et al., 2020) with two new linguistic annotation layers (POS and agreement chains), and explore to what extent different lexical categories and agreement phenomena are impacted by gender skews. Focusing on speech translation, we conduct a multifaceted evaluation on three language directions (English-French/Italian/Spanish), with models trained on varying amounts of data and different word segmentation techniques. By shedding light on model behaviours, gender bias, and its detection at several levels of granularity, our findings emphasize the value of dedicated analyses beyond aggregated overall results. As Matasovic (2004) posits:  Gender is perhaps the only grammatical category that ever evoked passion  and not only among linguists.  That is because, in the case of human entities, masculine or feminine inflections are assigned semantically, i.e. in relation to the extra-linguistic reality of gender (Ackerman, 2019; Corbett, 1991, 2013). Thus, gendered features interact with the  sociocultural and political  perception and representation of individuals (Gygax et al., 2019), by prompting discussions on the appropriate recognition of gender groups and their linguistic visibility (Stahlberg et al., 2007; Hellinger and Motschenbacher, 2015; Hord, 2016). Such concerns also invested language technologies (Sun et al., 2019; Cao and Daum III, 2020), where it has been shown that automatic translation systems tend to over-represent masculine forms and amplify stereotypes when translating into grammatical gender languages (Savoldi et al., 2021). Current evaluation practices for assessing gender bias in both Machine (MT) and Speech Translation (ST) commonly inspect such concerning behaviours by focusing only on a restricted set of occupational nouns (e.g. nurse , doctor ), and on synthetic benchmarks (Stanovsky et al., 2019; Escud Font and Costa-juss, 2019; Renduchintala et al., 2021). Also, even when relying on lexically richer natural benchmarks, the designed metrics still work at the word level, treating all gender-marked words indiscriminately (Alhafni et al., 2020; Bentivogli et al., 2020). Accordingly, current test sets and protocols: i) do not allow us to inspect if and to what extent different word categories participate in gender bias, ii) overlook the underlying morphosyntactic nature of grammatical gender on agreement chains, which cannot be monitored on single isolated words (e.g. en : a strange friend; it : una/o strana/o amica/o). In fact, to be grammatically correct, each word in the chain has to be inflected with the same (masculine or feminine) gender form. 1 We believe that fine-grained evaluations including the analysis of gender agreement across different parts of speech (POS) are relevant not only to gain a deeper understanding of bias in grammatical gender languages, but also to inform mitigating strategies and data curation procedures. Toward these goals, our contributions are as follows. (1) We enrich MuST-SHE (Bentivogli et al., 2020)  the only natural gender-sensitive benchmark available for MT and also ST  with two layers of linguistic information: POS and agreement 1 For an analogy, consider the case of (lack of) number agreement in the following: *a dogs barks. chains. 2 (2) In light of recent studies exploring how model design and overall perfomance interplay with gender bias (Roberts et al., 2020; Gaido et al., 2021), we rely on our manually curated resource to compare three ST models, which are trained on varying amounts of data, and built with different segmentation techniques: character and byte-pair-encoding (BPE) (Sennrich et al., 2016). We carry out a multifaceted evaluation that includes automatic and extensive manual analyses on three language pairs (en-es, en-fr, en-it) and we consistently find that: i) not all POS are equally impacted by gender bias; ii) translating words in agreement does not emerge as a systematic issue; iii) ST systems produce a considerable amount of neutral rewordings in lieu of gender-marked expressions, which current binary benchmarks fail to recognize. Finally, in line with concurring studies, we find that iv) character-based systems have an edge on translating gender phenomena, by favouring morphological and lexical diversity. While research in Natural Language Processing (NLP) initially prioritized narrow technical interventions to address the social impact of language technologies, we are recently attesting a shift toward a more comprehensive understanding of bias (Shah et al., 2020; Blodgett et al., 2020). Along this line, focus has been given to bias analysis in models' innards and outputs (Vig et al., 2020; Costa-juss et al., 2022), and to ascertain the validity of bias measurement practices (Blodgett et al., 2021; Antoniak and Mimno, 2021; Goldfarb-Tarrant et al., 2021). Complementary mounting evidence suggests that  rather than striving for generalizations  gender bias detection ought to incorporate contextual and linguistic specificity (Gonzlez et al., 2020; Ciora et al., 2021; Matthews et al., 2021; Malik et al., 2021; Kurpicz-Briki and Leoni, 2021), which however receives little attention due to a heavy focus on English NLP (Bender and Friedman, 2018). Purported agnostic approaches and evaluations (Bender, 2009) can prevent from drawing reliable conclusions and mitigating recommendations, as attested by monolingual studies on grammatical gender languages (Zhou et al., 2019; Gonen et al., 2019; Zmigrod et al., 2019) and in 2 The annotation layers are an extension of MuST-SHE v1.2 and are freely downloadable at: ict.fbk.eu/must-she/ under the same MuST-SHE licence (CC BY NC ND 4.0) Figure 1: Example of gender-mapping in translation from the parallel en-it portion of the natural MuST-SHE corpus. Unlike English, grammatical gender languages exhibit an elaborate morphological and syntactic system, where gender is overtly marked on numerous POS (e.g., verbs, determiners, nouns), and related words have to agree on the same gender features (see Figure 1 for an example). Still, current corpora and evaluation practices do not fully foreground systems' behaviour on such grammatical constraints. WinoMT (Stanovsky et al., 2019) represents the standard corpus to evaluate gender bias in MT within an English-to-grammatical gender language scenario. It has been progressively enriched with new features (Saunders et al., 2020; Kocmi et al., 2020), and adapted for ST (Costa-juss et al., 2020). While this resource can be useful to diagnose gender stereotyping at scale, it excludes languages' peculiarities since it is built on the concatenation of two corpora designed for English monolingual tasks 3  WinoGender (Rudinger et al., 2018) and WinoBias (Zhao et al., 2018)  which consist of synthetic sentences with the same structure and a pre-selected occupational lexicon (e.g. The lawyer yelled at the hairdresser because he did a bad job). 4 To increase variability, Troles and Schmid (2021) extend WinoBias by accompanying occupations with highly gender-stereotypical verbs 3 Gonzlez et al. (2020) note that the U.S. labor market statistics employed to define stereotypical associations are not always in line with other national gender statistics, thus they may impose an Anglo-centric frame for the detection of bias in other language scenarios. 4 Levy et al. (2021) recently created BUG on natural English data, but still it is limited to the evaluation of occupations. and adjectives. Their evaluation though, still only considers the translated professions as to verify if the co-occuring words might skew the models' assumptions. However, gender-marking involves also several other, so far less accounted POS categories, but if they are just as problematic is not clear yet. Existing bilingual (Alhafni et al., 2021), and multilingual (Bentivogli et al., 2020) natural benchmarks, instead, are manually curated as to identify a variety of gender phenomena specifically modeled on the accounted languages. As a result, they maximize lexical and contextual variability to inspect whether translation models yield feminine under-representation in real-world-like scenarios (Savoldi et al., 2021). However, since this variability is not mapped into fine-grained linguistic information, evaluations on such corpora do not single out which instances may be more responsible for gender bias. Finally, by considering each word in isolation, they neglect the underlying features of gender agreement, which determine the grammatical acceptability of the translation. To the best of our knowledge, only two works have currently interplayed issues of syntactic agreement and gender bias. Renduchintala and Williams (2021) designed a set of English sentences involving a syntactic construction that requires to translate an occupational term according to its unequivocal gender trigger (e.g. that nurse is a funny man). While they find that MT struggles even in such a simple setting, they only inspect the translation of a single disambiguated word ( nurse ) rather than a whole group of words in agreement. Closer to our intent, Gaido et al. (2020) analyze the output of different ST systems and note that their models seem to wrongly pick divergent gender inflections for unrelated words in the same sentence (e.g. en : As a researcher, professor; fr : En tant que chercheuse F , professeur M ) but not for dependency-related ones (e.g. en : The classic Asian student; it : [La classica studentessa asiatica] F ). Although limited in scope, their observation is worth being explored systematically. We thus conduct the very first study that intersects POS, agreement, and gender bias. In light of the above, a fine-grained evaluation of bias focused on POS and gender agreement requires the creation of a new dedicated resource. Rather than building it from scratch, we add two annotation layers to the existing MuST-SHE bench-PARTS-OF-SPEECH mark (Bentivogli et al., 2020), which is built on spoken language data retrieved from TED talks. Available for en-es/fr/it, it represents the only multilingual MT and STGBET 5 exhibiting a natural variety of gender phenomena, which are balanced across feminine and masculine forms. In the reference translations of the corpus, each target gender-marked word  corresponding to a neutral expression in the English source  is annotated with its alternative wrong gender form (e.g. en : the girl left ; it : la<il> ragazza  an-data<andato> via ). As further discussed in in Section 4.2, such a feature enables fine-grained analyses of gender realization, which can also disentangle systems' tendency to (over)generate masculine  over feminine forms  in translation. MuST-SHE thus allows the identification and pinpointed evaluation of numerous and qualitatively different grammatical gender instances under authentic conditions. Furthermore, the target languages covered in MuST-SHE (es, fr, it) are particularly suitable to focus on linguistic specificity. As a matter of fact, as Gygax et al. (2019) suggest, accounting for gender in languages with similar typological features allows for proper comparisons. 6 5 Gender Bias Evaluation Testset (Sun et al., 2019). 6 We underscore that our dedicated resources and experiments intentionally account for the specificities of three (com-parable) grammatical gender languages. Hence, we remain cautious of extending by default the results of our annotation and experiments to any other language. Parts-Of-Speech. We annotate each target gender-marked word in MuST-SHE with POS information. As shown in Table 1 ( a-c ), we differentiate among six POS categories: 7 i) articles, ii) pronouns, iii ) nouns, and iv ) verbs. For adjectives, we further distinguish v) limiting adjectives with mi-nor semantic import that determine e.g. possession, quantity, space ( my , some , this ); and vi) descriptive adjectives that convey attributes and qualities, e.g. glad , exhausted . This distinction enables to neatly sort our POS categories into the closed class of function words, or into the open one of content words (Schachter and Shopen, 2007). Since words from these two classes differ substantially in terms of variability, frequency, and semantics, we reckon they represent a relevant variable to account for in the evaluation of gender bias. Agreement. We also enrich MuST-SHE with linguistic information that is relevant to investigate the morphosyntactic nature of grammatical gender agreement. Gender agreement, or concord (Cor-bett, 2006; Comrie, 1999), requires that related words match the same gender form, as in the case of phrases , i.e. groups of words that constitute a single linguistic unit. 8 Thus, as shown in Table 1, we identify and annotate as agreement chains gender-marked words that constitute a phrase, such as a noun plus its modifiers ( d ), and verb phrases for compound tenses ( e ). Also, structures that involve a gender-marked (semi-) copula verb and its predicative complement are annotated as chains ( f ), although in such cases the agreement constraint is weaker. 9 This annotation lets us verify whether a model consistently picks the same gender paradigm for all words in the chain, enabling the assessment of its syntagmatic behaviour. POS and agreement annotation was manually carried out by 6 annotators (2 per language pair) undergoing a linguistics/translation studies MA degree, and with native/excellent proficiency in the assigned target language. For each language pair, 7 Some POS categories (e.g. conjunctions, adverbs) are not considered since they are not subject to gender inflection. matical e.g. es : *el M buen M nin F ( en: the good kid). 9 Such structure, due to the semantics of some linking verbs, can enable more flexibility. E.g. in French, Elle est devenue F un M canard M ( She became a duck ) is grammatical, although un canard (a duck) is formally masculine. they annotated the whole corpus independently, based on detailed guidelines (see Appendix A). For POS, we computed inter-annotator agreement (IAA) on label assignment with the kappa coefficient (in Scott's  formulation) (Scott, 1955). The resulting values of 0.92 (en-es), 0.94 (en-fr) and 0.96 (en-it) correspond to almost perfect agreement according to its standard interpretation (Lan-dis and Koch, 1977). For gender agreement, IAA was calculated on the exact match of the complete chains in the two annotations. The resulting Dice coefficients (Dice, 1945) of 89.23% (en-es), 93.0% (en-fr), and 94.34% (en-it), can be considered highly satisfactory given the more complex nature of this latter task. Except for few liminal cases that were excluded from the dataset, all disagreements were reconciled. We show the final annotation statistics in Table 2. Variations across languages are due to inherently cross-lingual differences. 10 While their discussion is beyond the scope of this work, overall these figures underscore the so far largely unaccounted variability of gender across lexical categories. Our experiments draw on studies exploring the relation between overall system performance, model size and gender bias. Vig et al. (2020) posit that bias increases with model size as larger systems better emulate biased training data. Working on WinoMT/ST, Kocmi et al. (2020) correlate higher BLEU scores and gender stereotyping, whereas Costa-juss et al. (2020) show that systems with lower performance tend to produce fewer feminine translations for occupations, but rely less on stereotypical cues. To account for these findings and inspect the behavior of different models under natural conditions, we experiment with three end-to-end 10 Spanish, for instance, relies less than French or Italian on the gender-enforcing to be auxiliary, resulting in less gender-marked verbs ( fr : est parti/ie; it :  partita/o; es : se ha ido). ST solutions, namely: LARGE-BPE , SMALL-BPE and SMALL-CHAR (see Appendix B for complete details about the models and training setups). Developed to achieve state-of-the-art performance, LARGE-BPE models rely on Transformer (Vaswani et al., 2017) and are trained in rich data conditions (1.25M ASR/ST utterances) by applying BPE segmentation (Sennrich et al., 2016). To achieve high performance, we made use of: i) all the available ST training corpora for the languages addressed, namely MuST-C (Cattoni et al., 2021) and Europarl-ST (Iranzo-Snchez et al., 2020); ii) consolidated data augmentation methods (Nguyen et al., 2020; Park et al., 2019; Jia et al., 2019); and iii) knowledge transfer techniques from ASR and MT, namely component pre-training and knowledge distillation (Weiss et al., 2017a; Bansal et al., 2019). 11 In terms of BLEU score  34.12 on en-es, 40.3 on en-fr, 27.7 on en-it  our LARGE-BPE models compare favorably with recently published results on MuST-C test data (Le et al. 2021 12 and Bentivogli et al. 2021 13 ). Also built with the same (Transformer-based) core technology, the other systems, SMALL-BPE and SMALL-CHAR , allow for apples-to-apples comparison between the different capabilities of BPE and character-level tokenization, namely: i) the syntactic advantage of BPE in managing several agreement phenomena (Sennrich, 2017; Ataman et al., 2019), and ii) the higher capability of character-level at generalizing morphology (Be-linkov et al., 2020). Given the morphological and syntactic nature of gender, such differences make them enticing candidates for further analysis. So far, Gaido et al. (2021) carried out the only study interplaying the two segmententation methods and gender bias, and found that  in spite of lower overall performance  character tokenization results in higher production of feminine forms for ST. By exploiting our new enriched resource, we intend to further test this finding and extend the analysis to gender agreement. Thus, for the sake of comparison with (Gaido et al., 2021), we train these systems in the same (controlled) data conditions i.e. on the MuST-C corpus only. 11 We are aware that both MuST-C and Europarl-ST are characterized by a majority (70%) of masculine speakers (Gaido et al., 2020; Vanmassenhove et al., 2018). Although comprehensive statistics are not available for the other ASR and MT training resources, we can reasonably assume they are similarly biased. 12 28.73 on en-es, 34.98 on en-fr, 24.96 on en-it. 13 32.93 on en-es, 28.56 on en-it. We employ the enriched MuST-SHE corpus to assess generic performance and gender translation at several levels of granularity. Evaluating gender translation under natural conditions grants the advantage of inspecting diverse informative phenomena. Concurrently, however, the intrinsic variability of natural language can defy automatic approaches based on reference translations: Since language generation is an open-ended task, in our specific setting system's outputs may not contain the exact gender-marked words annotated in MuST-SHE. In fact, the released MuST-SHE evaluation script (Gaido et al., 2020) first measures dataset coverage , i.e. the proportion of annotated words that are generated by the system, and on which gender translation is hence measurable. Then, it calculates gender accuracy as the proportion of words generated in the correct gender among the measurable ones. As a result, all the out of coverage words are necessarily left unevaluated. For all word-level gender evaluations (Sections 5.1 and 5.2), we compute accuracy as in the official MuST-SHE script and include scores based on the POS annotations. Instead, for chain-level gender agreement evaluation (Section 6.1) we modified the original script to process full agreement chains instead of single words. 14 Finally, since we aim at gaining qualitative insights into systems' behaviour, and at ensuring a sound and thorough multifaceted evaluation, we overcome the described coverage limitation of the automatic evaluation by complementing it with a manual analysis of all the gender-marked words and agreement chains that remained out of coverage. This extensive manual evaluation was carried out via a systematic annotation of systems' outputs, performed by the same linguists that enriched MuST-SHE, who provided the appropriate knowledge of both the resource and the evaluation task. Accordingly, we manage to make our study completely exhaustive by covering every gender-marked instance of MuST-SHE. Also, such additional manual evaluation serves as a proof-of-concept to ensure the validity of the employed automatic evaluation metrics. 14 The scripts are released together with the MuST-SHE annotated extensions. Table 3 presents SacreBLEU (Post, 2018), 15 coverage, and gender accuracy scores on the MuST-SHE test sets. All language directions exhibit a consistent trend: LARGE-BPE systems unsurprisingly achieve by far the highest overall translation quality. Also, in line with previous analyses (Di Gangi et al., 2020), SMALL-BPE models outperform the CHAR ones by  1 BLEU point. The higher overall translation quality of LARGE-BPE models is also reflected by the coverage scores (All-Cov), where they generate the highest number of MuST-SHE gender-marked words for all language pairs. By turning to overall gender accuracy (All-Acc) though, the edge previously assessed for the bigger state-of-the-art systems ceases to be clear-cut. For en-es and en-fr, LARGE-BPE systems outperform the concurring SMALL-CHAR by  2 points only  a slim advantage compared to the large gap observed on BLEU score. Moreover, for en-it, SMALL-CHAR proves the best at translating gender. We further zoom into the comparison of gender translation for feminine (F-Acc) and masculine (M-Acc) forms, where we can immediately assess that all ST models are skewed toward a disproportionate production of masculine forms (on average, 53.1% for F vs. 81.3% for M). However, focusing on LARGE-BPE models, we discover that their higher global gender accuracy (All-Acc) is actually due to the higher generation of masculine forms, while they do not compare favorably when it comes to feminine translation. In fact, in spite of achieving the lowest generic translation quality, SMALL-CHAR prove on par (for en-es) or even better (for en-it and en-fr) than LARGE-BPE models at handling feminine gender translation. tic metrics, are able to disentangle gender phenomena. As such, we can confirm that higher generic performance does not entail a superior capacity of producing feminine gender. This does not only emerge, as per Gaido et al. (2021), in the comparison of (small) BPEand char-based ST models. Rather, even for stronger systems, we attest how profiting from a wealth of  uncurated and synthetic (Bender et al., 2021)  data does not grant advantages to address gender bias. This motivates us to continue our multifaceted evaluation by taking into account only small models  henceforth CHAR and BPE  that, being trained on the same MuST-C data, allow for sound and transparent comparison. At a finer level of granularity, we use our extension of MuST-SHE to inspect gender bias across open and closed class words. Their coverage ranges between 74-81% for function words, but it shrinks to 44-59% for content words (see Appendix C.1). This is expected given the limited variability and high frequency of functional items in language. Instead, the coverage of feminine and masculine forms is on par within each class for all systems, thus allowing us to evaluate gender accuracy on a comparable proportion of generated words. A bird's-eye view of Figure 2 attests that, although masculine forms are always disproportionately produced, the gender accuracy gap is amplified on the open class words. The consistency of such a behaviour across languages and systems suggests that content words are involved to a greater extent in 1812 gender bias. We hence analyse this more problematic class by looking into a breakdown of the results per POS, while for function words' gender accuracy we refer to Appendix C.2. Table 4 presents results for verbs , nouns and descriptive adjectives . First, in terms of system capability, CHAR still consistently emerge as the favorite models for feminine translation. What we find notable, though, is that even within the same class we observe evident fluctuations, where nouns come forth as the most biased POS with a huge divide between M and F accuracy (5277 points). Specifically, scores below 50% indicate that feminine forms are generated with a probability that is below random choice, thus signalling an extremely strong bias. In light of this finding, we hypothesize that semantic and distributional features might be a factor to interpret words' gender skew. Specifically, occupational lexicon (e.g. lawyer, professor) makes up for most of the nouns represented in MuST-SHE (  70%). While such a high rate of professions in TED data is not surprising per se , 16 it singles out that professions may actually represent a category where systems largely rely on spurious cues to perform gender translation, even within natural conditions that do not ambiguously prompt stereotyping. We exclude basic token frequency by POS as a key factor to interpret our results, as MuST-SHE feminine nouns do not consistently appear as the POS with the lowest number of occurrences, nor do they have the lowest F:M ratio within MuST-C training data. As discussed in Section 8, we believe that our breakdown per POS is informative inasmuch it prompts qualitative considerations on how to pursue gender bias mitigation in models and corpora (Czarnowska et al., 2021; Doughman et al., 2021). We manually inspect CHAR and BPE system's output on the out-of-coverage (OOC) words that could not be automatically evaluated (see All-Cov column in Table 3), which amount to more than 5,000 instances. As shown in Table 5, our analysis discerns between OOC words due to i) translation errors (Err), 17 and ii) adequate alternative translations (i.e. meaning equivalent) for the expected gender-marked words. Such alternatives comprise instances in which word omission is acceptable 16 As TED talks are held by field experts, references to education and titles are quite common (MacKrill et al., 2021). (Alt-O) (Baker, 1992), and rewordings through synonyms or paraphrases. Since our focus remains on gender translation, we distinguish when such rewordings are generated with correct (Alt-C) or wrong (Alt-W) gender inflections, as well as neutral expressions devoid of gender-marking (Alt-N). Note that  with respect to English (Cao and Daum III, 2020; Vanmassenhove et al., 2021; Sun et al., 2021)  overcoming the structural pervasiveness of gender specifications in grammatical gender languages is extremely challenging (Gabriel et al., 2018a), but some rewordings can enable indirect neutral language (INL) 18 (Lpez, 2020). The results of the analysis are shown in Figure 3. Surprisingly, we find that BPE models  in spite of their higher BLEU scores  accumulate more translation errors than their CHAR counterparts. 19 Conversely, CHAR models generate an overall higher proportion of alternatives and, more importantly, alternatives whose gender translation is acceptable (-N, -C). This suggests that CHAR output is characterized by a favourable adequate variability that 18 INL relies on generic expressions rather than gender-specific ones (e.g. service vs. waiter/tress ) See Section 8. conveys both lexical meaning and gender realization better than BPE . Also, note that the outcome of the manual analyses reiterates the results obtained with the automatic evaluation based on accuracy at the word-level, thus confirming its reliability. As a final remark, we find that all systems produce a considerable amount of neutral alternatives in their outputs. To gain insight into such neutralizations, we audit on which POS they are realized. Accordingly, we find that neutralizations of adjectives and nouns are quite limited, and concern the production of epicene synonyms (e.g. en : happy; es-ref : contento/a; es-out : feliz). Verbs, instead, are largely implicated in the phenomenon, since inflectional changes in tense and aspect paradigms (e.g., present, imperfective) that do not convey gender distinctions are feasible (see the -N example in Table 5). Such range of alternatives for verbs is in fact also reflected by its lowest coverage among all POS (as low as  32%). Finally, paraphrases based on verbs also represent the most frequent way to neutralize other POS in the output. Since such expressions are suitable, or even preferable, for several scenarios (e.g. to substitute masculine generics, to avoid making unlicensed gender assumptions) our finding encourages the creation of test sets accounting for such a third viable direction, and can shed light on systems' potential to produce INL alternatives. The final step in our multifaceted analysis goes beyond the word level to inspect agreement chains in translation. To this aim, we define coverage as the proportion of generated chains matching with those annotated in MuST-SHE. Then, the accuracy of the generated chains accounts for 3 different cases where: i) agreement is respected, and with the correct gender (C); ii) agreement is respected, but with the wrong gender (W); and iii) both feminine and masculine gender inflections occur together, and thus agreement is not respected (NO). Table 6 shows accuracy scores for all MuST-SHE agreement chains (All), also split into feminine (F) and masculine (M) chains. The overall results are promising: we find very few instances (literally 1 or 2) in which ST systems produce an ungrammatical output that breaks gender agreement (NO). In fact, both systems tend to be consistent with one picked gender for the whole de-All Feminine Masculine C W NO C W NO C W NO en-es bpe 74.3 24.6 1.2 33.9 64.4 1.7 95.5 3.6 0.9 char 78.4 21.0 0.6 42.4 57.6 0.0 96.6 2.6 0.9 en-fr bpe 67.9 31.0 1.2 54.1 45.9 0.0 78.7 19.1 2.1 char 76.7 22.3 1.0 57.5 40.0 2.5 88.9 11.1 0.0 en-it bpe 71.7 27.5 0.7 47.4 50.9 1.8 88.9 11.1 0.0 char 78.5 20.0 1.5 54.2 44.1 1.7 97.4 1.3 1.3 Table 6: Agreement results for All chains matched in MuST-SHE, and split into Feminine and Masculine chains. pendency group. Thus, in spite of previous MT studies concluding that character-based segmentation results in poorer syntactic capability (Belinkov et al., 2020), respecting concord does not appear as an issue for any of our small ST models. For the sake of comparability, however, we note that our evaluation involves language pairs that do not widely resort to long-range dependencies; this may contribute to explaining why CHAR better handles correct gender agreement. 20 Overall, agreement translation was measured on a lower coverage (30-50%)  presented in Appendix D.1  than the word-level one (Section 3). While this is expected given the strict requirement of generating full chains with several words, we recover such a loss by means of the comprehensive manual evaluation discussed below. Our manual inspection recovers a total of  1,200 OOC agreement chains from CHAR and BPE output. Similarly to the approach employed for single words (Section 5.3), we discern between OOC chains due to: i) translation errors (Err), and ii) alternative translations preserving the source meaning. We distinguish different types of alternatives. First, alternatives that do no exhibit a morphosyntactic agreement phenomenon to be judged, as in the case of neutral paraphrases or rewordings consisting of a single word (NO-chain). Instead, when the generated alternative chain exhibits gender markings, we distinguish if the chosen gender is correct (C), wrong (W), or if the system produces a chain that does not respect gender agreement because it combines both feminine and masculine gender inflections (NO). 20 Due to space constraints we refer to Appendix D.2 for an analysis of longer-range cases of subject-verb agreement. is presented in Figure 4. Interestingly, such results are only partially corroborating previous analyses. On the one hand, unlike the OOC words' results discussed in Section 5.3, we attest that CHAR models produce the highest proportion of translation errors. Thus, it seems that CHAR capability in producing adequate alternatives is confined to the single-word level, whereas it exhibits a higher failure rate on longer sequences. On the other hand, by looking at alternative chains, CHAR still emerges as the best at properly translating gender agreement, with the highest proportion of chains with correct gender (C), and the lowest one with wrong gender (W). Finally, again in line with our automatic evaluation (Table 6), we confirm that respecting agreement is not an issue for our ST models: we identify only 3 cases (2 for en-fr BPE , 1 for en-fr CHAR ) where concord is broken (NO). Given the rarity of such instances, we are not able to draw definitive conclusions on the nature of these outliers. Nonetheless, we check the instances in which agreement was not respected (both in and out of coverage). We see that cases of broken concord also concern extremely simple phrases, consisting of a noun and its modifier (e.g. en: talking to [this inventor],...because he ; fr: parler  [cette F inventeur M ] ..., parce qu' il ). However, the most common type among these outliers are constructions with semi-copula verbs (e.g. en: She... [became a vet] ; it: ...E' [ diventata F un M veterinatrio M ] ), which  as discussed in Section 3.1  exhibit a weaker agreement constraint. The complex system of grammatical gender languages entails several morphosyntactic implications for different lexical categories. In this paper, we underscored such implications and explored how different POS and grammatical agreement are involved in gender bias. To this aim, we enriched the MuST-SHE benchmark with new linguistic information, and carried out an extensive evaluation on the behaviour of ST models built with different segmentation techniques and data quantities. On three language pairs (English-French/Italian/Spanish), our study shows that, while all POS are subject to masculine skews, they are not impacted to the same extent. Respecting gender agreement for the translation of related words, instead, is not an issue for current ST models. We also find that ST generates a considerable amount of neutral expressions, suitable to replace gender-inflected ones, which however current test sets do not recognize. Overall, our work reiterates the importance of dedicated analyses that, unlike holistic metrics, can single out system's behaviour on gender phenomena. Accordingly, our results are in line with previous studies showing that, in spite of lower generic performance, character-based segmentation exhibits a better capability at handling feminine translation at different levels of granularity. As our MuST-SHE extension is available for both ST and MT, we invite MT studies to start from our discoveries and resource. We would like to thank the 2021 Summer Internship students at FBK for their contribution: Francesco Fernicola, Sara Giuliani, Lorena Rocio Martn, Silvia Alma Piazzolla, Mlanie Prati, Jana Waldmann. This work was made possible thanks to their extensive annotation work and active participation in fruitful discussions. In this paper, we evaluate whether and to what extent ST models exhibit biased behaviors by systematically and disproportionately favoring masculine forms in translation. Such a behavior is problematic inasmuch it leads to under-representational harms by reducing feminine visibility (Blodgett et al., 2020; Savoldi et al., 2021). Broader impact. While the focus of this work is on the analysis itself, our insights prompt broader considerations. Specifically, our investigation on the relation between data size/segmentation technique and gender bias provides initial cues on which models and components to audit and implement toward the goal of reducing gender bias. This, in particular, may be informative to define the path for emerging 1815 direct ST technologies. Also, our results disaggregated by POS invite reflections on how to intend and mitigate bias by means of interventions on the training data. In fact, while it is known that the MuST-C corpus (Cattoni et al., 2021) used for training comprises a majority of masculine speakers, 21 the fact that certain lexical categories are more biased than others suggests that, on top of more coarse-grained quantitative attempts at gender balancing (Costa-juss and de Jorge, 2020), data curation ought to account for more sensitive, nuanced, and qualitative asymmetries. These also imply how , rather than only how often , gender groups are represented (Wagner et al., 2015; Devinney et al., 2020). Also, while nouns come forth as the most problematic POS, current practices of data augmentation based on a pre-defined occupational lexicon may address stereotyping (Saunders and Byrne, 2020), but do not increase the production of other nonetheless skewed lexical categories. Overall, our enriched resource 22 can be useful to monitor the validity of different technical interventions. Ethic statement. The use of gender as a variable (Larson, 2017) warrants some ethical reflections. Our evaluation on the MuST-SHE benchmark exclusively accounts for linguistic gender expressions. As reported in MuST-SHE data statement (Bender and Friedman, 2018), 23 also for the subset of sentences that contain first-person references 24 (e.g. I'm a student ), speakers' gender information is manually annotated based on the personal pronouns found in their publicly available personal TED profile, and used to check that the indicated (English) linguistic gender forms are rendered in the gold standard translations. While our experiments are limited to the binary linguistic forms represented in the used data, to the best of our knowledge, ST natural language corpora going beyond binarism do not yet exist. 25 This is also due to the fact that unlike English  which finds itself for several cultural and linguistic reasons as a leader of change toward inclusive forms (Ackerman, 2019)  Direct Nonbinary Language based on neomorphemes (Shroy, 21 https://ict.fbk.eu/must-speakers/ 22 It will be released under the same CC BY NC ND 4.0 International license as MuST-SHE. Category 1 in the corpus. 25 Saunders et al. (2020) enriched WinoMT to account for non-binary language. While it is only available for MT, such annotations consist of placeholders for neutrality rather than actual non-binary expressions. 2016; Papadopoulos, 2019; Knisely, 2020) is nontrivial to fully implement in grammatical gender languages (Hellinger and Buman, 2001; Gabriel et al., 2018b) and still object of experimentation (Redazione, 2020; Attig and Lpez, 2020). However, our manual evaluation expands to the possibility of INL strategies that could be detected in system's output. We underscore that such strategies are recommended and fruitful to avoid the gendering of referents, but are to be considered as concurring to  rather than replacements of  emerging linguistic innovations (Lpez, 2020). Lastly, we signal that direct ST models may leverage speakers' vocal characteristics as a gender cue to infer gender translation. Although the potential risks of such condition do not emerge and are not addressed in our setting (focused on POS and agreement features as a variable), we endorse the point made by Gaido et al. (2020). Namely, direct ST systems leveraging speaker's vocal biometric features as a gender cue can entail real-world dangers, like the categorization of individuals by means of biological essentialist frameworks (Zim-man, 2020). This can reduce gender to stereotypical expectations about how masculine or feminine voices should sound, and can be especially harmful to transgender individuals, as it can lead to misgen-dering (Stryker, 2008) and invalidation. Note that we experimented with unmodified models for the sake of hypothesis testing without adding variability, but real-world deployment of ST technologies must account for the potential harms arising form the use of direct ST technologies as is .
abstain abstain abstain abstain method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective method abstain other result abstain abstain abstain result abstain abstain other other other other other other other other other other other other other other other other other abstain other other method other objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective objective result abstain result abstain result abstain other other method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other method other abstain abstain abstain abstain abstain method abstain method abstain abstain method