We present a benchmark suite of four datasets for evaluating the fairness of pre-trained language models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (Euro-pean Council, USA, Switzerland, and China), five languages (English, German, French, Italian and Chinese) and fairness across five attributes (gender, age, region, language, and legal area). In our experiments, we evaluate pre-trained language models using several group-robust fine-tuning techniques and show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Furthermore, we provide a quantitative and qualitative analysis of our results, highlighting open challenges in the development of robustness methods in legal NLP. Natural Language Processing (NLP) for law (Chalkidis and Kampas, 2019; Aletras et al., 2019; Zhong et al., 2020; Chalkidis et al., 2022) receives increasing attention. Assistive technologies can speed up legal research or discovery significantly assisting lawyers, judges and clerks. They can also help legal scholars to study case law (Katz, 2012; Coupette et al., 2021), improve access of law to laypersons, help sociologists and research ethicists to expose biases in the justice system (Angwin et al., 2016; Dressel and Farid, 2018), and even scrutinize decision-making itself (Bell et al., 2021). In the context of law, the principle of equality and non-discrimination is of paramount importance, although its definition varies at international, regional and domestic level. For example, EU nondiscrimination law prohibits both direct and indirect discrimination. Direct discrimination occurs when one person is treated less favourably than  Corresponding author: ilias.chalkidis@di.ku.dk Figure 1: Group disparity for defendant state (C.E. Europe vs. The Rest) in ECtHR and legal area (Civil law vs. Penal law) in FSCS. others would be treated in comparable situations on grounds of sex, racial or ethnic origin, disability, sexual orientation, religion or belief and age. 1 Given the gravity that legal outcomes have for individuals, assistive technologies cannot be adopted to speed up legal research at the expense of fairness (Wachter et al., 2021), potentially also decreasing the trust in our legal systems (Barfield, 2020). Societal transformations perpetually shape our legal systems. The topic deserves great attention because AI systems learning from historical data pose the risk of lack of generalisability beyond the training data, and more importantly transporting biases previously encumbered in the data in future decision-making, thereby exponentially increasing their e  ect (Delacroix, 2022). Historical legal data do not represent all groups in our societies equally and tend to reflect social biases in our societies and legal institutions. When models are deployed in production, they may reinforce these biases. For example, criminal justice is already often strongly influenced by racial bias, with people of colour being more likely to be arrested and receive higher punishments than others, both in the USA 2 and in the UK. 3 1 An in-depth analysis of the notion of discrimination and fairness in law is presented in Appendix A. 2 https://tinyurl.com/4cse552t 3 https://tinyurl.com/hkff3zcb 4389 In recent years, the NLP and machine learning literature has introduced fairness objectives, typically derived from the Rawlsian notion of equal opportunities (Rawls, 1971), to evaluate the extent to which models discriminate across protected attributes. Some of these rely on notions of resource allocation, i.e., reflecting the idea that groups are treated fairly if they are equally represented in the training data used to induce our models, or if the same number of training iterations is performed per group. This is sometimes referred to as the resource allocation perspective on fairness (Lundgard, 2020). Contrary, there is also a capability -centered approach to fairness (Anderson, 1999; Robeyns, 2009), in which the goal is to reserve enough resources per group to achieve similar performance levels, which is ultimately what is important for how individuals are treated in legal processes. We adopt a capability-centered approach to fairness and define fairness in terms of performance parity (Hashimoto et al., 2018) or equal risk (Donini et al., 2018). 4 Performance disparity (Hashimoto et al., 2018) refers to the phenomenon of high overall performance, but low performance on minority groups, as a result of minimizing risk across samples (not groups). Since some groups benefit more than others from models and technologies that exhibit performance disparity, this likely widens gaps between those groups. Performance disparity works against the ideal of fair and equal opportunities for all groups in our societies. We therefore define a fair classifier as one that has similar performance (equal risk) across all groups (Donini et al., 2018). In sum, we adopt the view that (approximate) equality under the law in a modern world requires that our NLP technologies exhibit (approximately) equal risk across sensitive attributes. For everyone to be treated equally under the law, regardless of race, gender, nationality, or other characteristics, NLP assistive technologies need to be (approxi-mately) insensitive to these attributes. We consider three types of attributes in this work:  Demographics : The first category includes demographic information of the involved parties, e.g., the gender, sexual orientation, nationality, age, or race of the plainti / defendant in a case. In this case, we aim to mitigate biases against specific 4 The dominant alternative to equal risk is to define fairness in terms of equal odds . groups, e.g., a model performs worse for female defendants or is biased against black defendants. We can further consider information involving the legal status of involved parties, e.g., person vs. company, or private vs. public. Regional : The second category includes regional information, for example the courts in charge of a case. In this case, we aim to mitigate disparity in-between di  erent regions in a given jurisdiction, e.g., a model performs better in specific cases originated or ruled in courts of specific regions. Legal Topic : The third category includes legal topic information on the subject matter of the controversy. In this case, we aim to mitigate disparity in-between di  erent topics (areas) of law, e.g., a model performs better in a specific field of law, for example penal cases. Contributions We introduce FairLex, a multilingual fairness benchmark of four legal datasets covering four jurisdictions (Council of Europe, United States of America, Swiss Confederation and People's Republic of China), five languages (English, German, French, Italian and Chinese) and various sensitive attributes (gender, age, region, etc.). We release four pre-trained transformer-based language models, each tailored for a specific dataset (task) within our benchmark, which can be used as baseline models (text encoders). We conduct experiments with several group-robust algorithms and provide a quantitative and qualitative analysis of our results, highlighting open challenges in the development of robustness methods in legal NLP. Fair machine learning The literature on inducing approximately fair models from biased data is rapidly growing. See Mehrabi et al. (2021); Makhlouf et al. (2021); Ding et al. (2021) for recent surveys. We rely on this literature in how we define fairness, and for the algorithms that we compare in our experiments below. As already discussed, we adopt a capability-centered approach to fairness and define fairness in terms of performance parity (Hashimoto et al., 2018) or equal risk (Donini et al., 2018). The fairness-promoting learning algorithms we evaluate are discussed in detail in Section 4. Some of these  Group Distributionally Robust Optimization (Sagawa et al., 2020) and Invariant Risk Minimization (Arjovsky et al., 2020)  have previously been evaluated for fairness in the context of hate speech (Koh et al., 2021). Fairness in law Studying fair machine learning in the context of legal (computational) applications has a limited history. In a classic study, Angwin et al. (2016) analyzed the performance of the Correctional O  ender Management Profiling for Alternative Sanctions (COMPAS) system, which was used for parole risk assessment (recidivism prediction) in the US. The system relied on 137 features from questionnaires and criminal records. Angwin et al. found that blacks were almost twice as likely as whites to be mislabeled as high risk (of re-o  ending), revealing a severe racial bias in the system. The system was later compared to crowd-workers in Dressel and Farid (2018). These studies relied on tabular data and did not involve text processing (e.g., encoding case facts or decisions). More recently, Wang et al. (2021b) studied legal judgment consistency using a dataset of Chinese criminal cases. They evaluated the consistency of LSTM-based models across region and gender and reported severe fairness gaps across gender. They also found that the fairness gap was particular severe for more serious crimes. Another line of work (Rice et al., 2019; Baker Gillis, 2021; Gu-musel et al., 2022) explores representational bias with respect to race and gender analyzing word latent representations trained in legal text corpora. While we agree that representational bias can potentially reinforce unfortunate biases, these may not impact the treatment of individuals (or groups). We therefore focus on directly measuring equal risk on downstream applications instead. Previous work has focused on the analysis of specific cases, languages or algorithms, but FairLex aims at easing the development and testing of bias-mitigation models or algorithms within the legal domain. FairLex allows researchers to explore fairness across four datasets covering four jurisdictions (Council of Europe, United States of America, Swiss Confederation and People's Republic of China), five languages (English, German, French, Italian and Chinese) and various sensitive attributes (gender, age, region, etc.). Furthermore, we provide competitive baselines including pre-trained transformer-based language models, adapted to the examined datasets, and an in-dept examination of performance of four group robust algorithms described in detail in Section 4. Benchmarking NLP has been stormed by the rapid development of benchmark datasets that aim to evaluate the performance of pre-trained language models with respect to di  erent objectives: general Natural Language Understanding (NLU) (Wang et al., 2019b,a), Cross-Lingual Transfer (CLT) (Hu et al., 2020), and even domain-specific ones on biomedical (Peng et al., 2019), or legal (Chalkidis et al., 2022) NLP tasks. Despite their value, recent work has raised criticism on several limitations of the so called NLU benchmarks (Paullada et al., 2020; Bowman and Dahl, 2021; Raji et al., 2021). The main points are: poor ( laissez-faire ) dataset development (e.g., lack of diversity, spurious correlations), legal issues (e.g., data licensing and leakage of personal information), construct validity (e.g., poor experimental setup, unclear research questions), question of general capabilities, and promotion of superficial competitiveness (hype, or even falsify, state-of-the-art results). We believe that the release of FairLex, a domain-specific (legal-oriented) benchmark suite for evaluating fairness, overcomes (or at least mitigates) some of the aforementioned limitations. We introduce the core motivation in Section 1, while specific (case-by-case) details are described in Section 3. Our benchmark is open-ended and inevitably has several limitations; we report known limitations and ethical considerations in Sections 7 and 8. Nonetheless we believe that it will help critical research in the area of fairness. ECtHR The European Court of Human Rights (ECtHR) hears allegations that a state has breached human rights provisions of the European Convention of Human Rights (ECHR). We use the dataset of Chalkidis et al. (2021), which contains 11K cases from ECtHR's public database. Each case is mapped to articles of the ECHR that were violated (if any). This is a multi-label text classification task. Given the facts of a case, the goal is to predict the ECHR articles that were violated, if any, as decided (ruled) by the court. The cases are chronologically split into training (9k, 200116), development (1k, 201617), and test (1k, 201719) sets. To facilitate the study of fairness of text classifiers, we record for each case the following attributes: (a) The defendant states , which are the European states that allegedly violated the ECHR. The defendant states for each case is a subset of the 47 Member States of the Council of Europe; 5 To have statistical support, we group defendant states 5 https://www.coe.int/ 4391 Dataset Original Publication Classification Task No of Classes Attributes Attribute Type #N ECtHR (Chalkidis et al., 2021) Legal Judgment Prediction: ECHR Violation Prediction 10 + 1 Defendant State 2 Applicant Gender 2 Applicant Age 3 SCOTUS (Spaeth et al., 2020) Legal Topic Classification: Issue Area Classification 14 Respondent Type 4 Decision Direction 2 FSCS (Niklaus et al., 2021) Legal Judgment Prediction: Case Approval Prediction 2 Language 3 Region of Origin 6 Legal Area 6 CAIL (Wang et al., 2021b) Legal Judgment Prediction: Crime Severity Prediction 6 Defendant Gender 2 Region of Origin 7 Table 1: Main characteristics of FairLex datasets (ECtHR, SCOTUS, FSCS, CAIL). in two: Central-Eastern European states, on one hand, and all other states, as classified by the EuroVoc thesaurus. (b) The applicant's age at the time of the decision. We extract the birth year of the applicant from the case facts, if possible, and classify its case in an age group (  35,  64, or older) ; and (c) the applicant's gender , extracted from the facts, if possible, based on pronouns or other gendered words, classified in two categories (male, female). 6 SCOTUS The US Supreme Court (SCOTUS) is the highest federal court in the United States of America and generally hears only the most controversial or otherwise complex cases which have not been su  ciently well solved by lower courts. We combine information from SCOTUS opinions with the Supreme Court DataBase (SCDB) 7 (Spaeth et al., 2020). SCDB provides metadata (e.g., date of publication, decisions, issues, decision directions and many more) for all cases. We consider the available 14 thematic issue areas (e.g, Criminal Procedure, Civil Rights, Economic Activity, etc.) as labels. This is a single-label multi-class document classification task. Given the court opinion, the goal is to predict the issue area whose focus is on the subject matter of the controversy (dis-pute). SCOTUS contains a total of 9,262 cases that we split chronologically into 80% for training (7.4k, 19461982), 10% for development (914, 19821991) and 10% for testing (931, 19912016). From SCDB, we also use the following attributes to study fairness: (a) the type of respondent , which is a manual categorization of respondents (defen-dants) in five categories (person, public entity, organization, facility and other); and (c) the direction of the decision , i.e., whether the decision is considered liberal, or conservative, provided by SCDB. FSCS The Federal Supreme Court of Switzerland (FSCS) is the last level of appeal in Switzerland and similarly to SCOTUS, the court generally hears only the most controversial or otherwise complex cases which have not been su  ciently well solved by lower courts. The court often focus only on small parts of previous decision, where they discuss possible wrong reasoning by the lower court. The Swiss-Judgment-Predict dataset (Niklaus et al., 2021) contains more than 85K decisions from the FSCS written in one of three languages (50K German, 31K French, 4K Italian) from the years 2000 to 2020. The dataset provides labels for a sim-plified binary ( approval , dismissal ) classification task. Given the facts of the case, the goal is to predict if the plainti  's request is valid or partially valid. The cases are also chronologically split into training (59.7k, 2000-2014), development (8.2k, 2015-2016), and test (17.4k, 2017-2020) sets. The original dataset provides three additional attributes: (a) the language of the FSCS written decision, in either German, French, or Italian; (b) the legal area of the case (e.g., public, penal law) derived from the chambers where the decisions were heard; and (c) the region that denotes in which federal region was the case originated. CAIL The Supreme People's Court of China is the last level of appeal in China and considers cases that originated from the high people's courts concerning matters of national importance. The Chinese AI and Law challenge (CAIL) dataset (Xiao et al., 2018) is a Chinese legal NLP dataset for judgment prediction and contains over 1m criminal cases. The dataset provides labels for relevant article of criminal code prediction, charge (type of crime) prediction, imprisonment term (period) prediction, and monetary penalty prediction. 8 8 The publication of the original dataset has been the topic of an active debate in the NLP community (Leins et al., 2020; Tsarapatsanis and Aletras, 2021; Bender, 2021). Recently, Wang et al. (2021b) re-annotated a subset of approx. 100k cases with demographic attributes. Specifically the new dataset has been annotated with: (a) the applicant's gender , classified in two categories (male, female); and (b) the region of the court that denotes in which out of the 7 provincial-level administrative regions was the case judged. We re-split the dataset chronologically into training (80k, 2013-2017), development (12k, 2017-2018), and test (12k, 2018) sets. In our study, we re-frame the imprisonment term prediction and examine a soft version, dubbed crime severity prediction task, a multi-class classification task, where given the facts of a case, the goal is to predict how severe was the committed crime with respect to the imprisonment term. We approximate crime severity by the length of imprisonment term, split in 6 clusters (0,  12,  36,  60,  120, > 120 months). Across experiments, our main goal is to find a hypothesis for which the risk R ( h ) is minimal: h  = arg min h H R ( h ) (1) R ( h ) = E ( L ( h ( x ) , y )) (2) Similar to previous studies, R ( h ) is an expectation of the selected loss function ( L ). In this work, we study multi-label text classification (Section 3), thus we aim to minimize the binary cross-entropy loss across L classes: L =  y log  y  (1  y ) log(1   y ) (3) ERM (Vapnik, 1992), which stands for Empirical Risk Minimization, is the most standard and widely used optimization technique to train neural methods. The loss is calculated as follows: LERM = N (cid:88) i = 1 L i N (4) where N is the number of instances (training examples) in a batch, and L i is the loss per instance. Besides ERM, we also consider a representative selection of group-robust fine-tuning algorithms which aims at mitigating performance disparities with respect to a given attribute ( A ), e.g., the gender of the applicant or the region of the court. Each attribute is split into G groups, i.e., male / female for gender. All algorithms rely on a balanced group sampler, i.e., an equal number of instances (sam-ples) per group ( NG ) are included in each batch. Most of the algorithms are built upon group-wise losses ( L g ), computed as follows: L ( g i ) = 1 N g i N gi (cid:88) j = 1 L ( x j ) (5) Group DRO (Sagawa et al., 2020), stands for Group Distributionally Robust Optimization (DRO). Group DRO is an extension of the Group Uniform algorithm, where the group-wise losses are weighted inversely proportional to the group training performance. The total loss is: LDRO = G (cid:88) i = 1 w g i  L ( g i ), where (6) w g i = 1 W (  w g i  e L ( g i ) ) and W = G (cid:88) i = 1 w g i (7) where G is the number of groups (labels), L g are the averaged group-wise (label-wise) losses, w g are the group (label) weights,  w g are the group (label) weights as computed in the previous update step. Initially the weight mass in equally distributed across groups. V-REx (Krueger et al., 2020), which stands for Risk Extrapolation, is yet another proposed group-robust optimization algorithm. Krueger et al. (2020) hypothesize that variation across training groups is representative of the variation later encountered at test time, so they also consider the variance across the group-wise losses. In V-REx the total loss is calculated as follows: LREX = LERM +   Var([ L g 1 , . . . , L g G ]) (8) where Var is the variance among the group-wise losses and  , a weighting hyper-parameter scalar. IRM (Arjovsky et al., 2020), which stands for Invariant Risk Minimization, mainly aims to penalize variance across multiple training dummy estimators across groups, i.e., performance cannot vary in samples that correspond to the same group. The total loss is computed as follows: LIRM = 1 G   G (cid:88) i = 1 L ( g i ) +   P ( g i )   (9) Please refer to Arjovsky et al. (2020) for the definition of the group penalty terms ( P g ). Adversarial Removal (Elazar and Goldberg, 2018) algorithm mitigates group disparities by means of an additional adversarial classifier (Good-fellow et al., 2014). The adversarial classifier share the encoder with the main network and is trained to predict the protected attribute ( A ) of an instance. The total loss factors in the adversarial one, thus penalizing the model when it is able to discriminate groups. Formally, the total loss is calculated as: LAR = LERM    LADV (10) LADV = L ( g i , g i ) (11) where  g i is the adversarial classifier's prediction for the examined attribute A (in which group ( g i ) of A , does the example belong to) given the input ( x ). Models Since we are interested in classifying long documents (up to 6000 tokens per document, see Figure 2 in Appendix E.1), we use a hierarchical BERT-based model similar to that of Chalkidis et al. (2021), so as to avoid using only the first 512 tokens of a text. The hierarchical model, first, encodes the text through a pre-trained Transformer-based model, thus representing each paragraph independently with the [CLS] token. Then, the paragraph representations are fed into a two-layers transformer encoder with the exact same specifica-tions of the first one (e.g., hidden units, number of attention heads), so as to contextualize them, i.e., it makes paragraphs representations aware of the surrounding paragraphs. Finally, the model max-pools the context-aware paragraph representations computing the document-level representation and feed it to a classification layer. For the purpose of this work, we release four domain-specific BERT models with continued pretraining on the corpora of the examined datasets. 9 We train mini-sized BERT models with 6 Transformer blocks, 384 hidden units, and 12 attention heads. We warm-start all models from the public MiniLMv2 models checkpoints (Wang et al., 2021a) using the distilled version of RoBERTa (Liu et al., 2019) for the English datasets (ECtHR, SCOTUS) and the one distilled from XLM-R (Conneau et al., 2020) for the rest (trilingual FSCS, and Chinese CAIL). Given the limited size of these models, we can e  ectively use up to 4096 tokens in ECtHR and SCOTUS and up to 2048 tokens in FSCS and 9 https://huggingface.co/coastalcph CAIL for up to 16 samples per batch in a 24GB GPU card. 10 For completeness, we also consider linear Bag-of Words (BoW) classifiers using TF-IDF scores of the most frequent n -grams (where n = 1 , 2 , 3) in the training corpus of each dataset. Data Repository and Code We release a uni-fied version of the benchmark on Hugging Face Datasets (Lhoest et al., 2021). 11 In our experiments, we use and extend the WILDs (Koh et al., 2021) library. For reproducibility and further exploration with new group-robust methods, we release our code on Github. 12 Evaluation Details Across experiments we compute the macro-F1 score per group (mF1 i ), excluding the group of unidentified instances, if any. 13 We report macro-F1 to avoid bias toward majority classes because of class imbalance and skewed label distributions across train, development, and test subsets (Sgaard et al., 2021). Main Results In Table 2, we report the group performance (mF1), where models trained with the ERM algorithm, across all datasets and attributes. We observe that the intensity of group disparities vary a lot between di  erent attributes, but in many cases the group disparities are very vibrant. For example, in ECtHR, we observe substantial group disparity between the two defendant state groups (21.5% absolute di  erence), similarly for applicant's gender groups (16.2% absolute di  er-ence). In FSCS, we observe language disparity, where performance is on average 3-5% lower for cases written in Italian compared to those written in French and German. Performance disparity is even higher with respect to legal areas , where the model has the best performance for criminal (penal law) cases (83.4%) compared to others (approx. 10-20% lower). We also observe substantial group disparities with respect to the court region , e.g., cases ruled in E. Switzerland courts (66.8%) compared to Federation courts (56.4%). The same applies for CAIL, e.g., cases ruled in Beijing courts (66.8%) compared to Sichuan courts (56.4%). Group Disparity Analysis Moving forward we try to identify general (attribute agnostic) factors based on data distributions that could potentially lead to performance disparity across groups. We identify three general (attribute agnostic) factors:  Representation Inequality : Not all groups are equally represented in the training set. To examine this aspect, we report the number of training cases per group. Temporal Concept Drift : The label distribution for a given group changes over time, i.e., in-between training and test subsets. To examine this aspect, we report per group, the KL divergence in-between the training and test label distribution. Worst Class Influence : The performance is not equal across labels (classes), which may disproportionally a  ect the macro-averaged performance across groups. To examine this aspect, we report the Worst Class Influence (WCI) score per group, which is computed as follows: WCI( i ) = #test-cases (worst-class) #test-cases (12) In Table 2, we present the results across all attributes. We observe that only in 4 out of 10 cases (attributes), the less represented groups are those with the worst performance compared to the rest. It is generally not the case that high KL divergence (drift) correlates with low performance. In other words, group disparities does not seem to be driven by temporal concept drift. Finally, the influence of the worst class is relatively uniform across groups in most cases, but in the cases where groups di  er in this regard, worst class influence correlates with error in 2 out of 3 cases. 14 In ECtHR, considering performance across defendant state, we see that all the three factors correlate internally, i.e., the worst performing group is less represented, has higher temporal drift and has more cases in the worst performing class. This is not the case considering performance across other attributes. It is also not the case for SCOTUS. In FSCS, considering the attributes of language and region, representation inequality seems to be an important factor that leads to group disparity. This is not the case for legal area, where the best 14 For ECtHR performance across defendant states and SCOTUS across directions, but not for ECtHR performance across applicant age. represented group is the worst performing group. In other words, there are other reasons that lead to performance disparity in this case; according to Niklaus et al. (2021), a potential factor is that the jurisprudence in penal law is more united and aligned in Switzerland and outlier judgments are rarer making the task more predictable. Cross-Attribute Influence Analysis We have evaluated fairness across attributes that are not necessarily independent of each other. We therefore evaluate the extent to which performance disparities along di  erent attributes correlate, i.e., how attributes interact, and whether performance di  erences for attribute A 1 can potentially explain performance di  erences for another attribute A 2 . We examine this for the two attributes with the highest group disparity: the defendant state in ECtHR, and the legal area in FSCS. For the bins induced by these two attributes ( A 1 ), we compute mF1 scores across other attributes ( A 2 ). In ECtHR, approx. 83% and 81% of male and women applicants are involved in cases against E.C. European states (best-performing group). Similarly, in case of age groups, we observe that ratio of cases against E.C. European states is: 87% and 86% for  65 and  35, the bestand worst-performing groups respectively. In FSCS, the ratio of cases relevant to penal law is: approx. 29%, and 41% written in written in French (best-performing group) and Italian (worst-performing group). Similarly, approx. 27% originated in E. Switzerland (best-performing group) and 42% in Federation (worst performing group) are relevant to public law. In both attributes, there is a 15% increase of cases relevant to public law for the worst performing groups. In other words, the group disparity in one attribute A 2 (language, region) could be also explained by the influence of another attribute A 1 (legal area). In Table 3, we report the performance in the aforementioned cross-attribute ( A 1 , A 2 ) pairings. With the exception of the (age, defendant state) cross-examination in ECtHR, we observe that group disparities in attribute A 2 (Table 2) are consistent across groups of the plausible influencer (i.e. attribute A 1 ). Hence, cross-attribute influence does not explain the observed group disparities. We believe that such an in-depth analysis of the results is fundamental to understand the influence of di  erent factors in the outcomes. This analysis wouldn't be possible, if we had counterfeited an ideal scenario, where all groups and labels where equally represented. While a controlled experimental environment is frequently used to examine specific factors, it could hide, or partially alleviate such phenomena, hence producing misleading results on fairness of the examined models. Group Robust Algorithms Results Finally, we evaluate the performance for several group robust algorithms ( Section 4) that could potentially mitigate group disparities. To estimate their performance, we report the average macro-F1 across groups (mF1) and the group disparity (GD) among groups measured as the group-wise std dev.: GD = (cid:118)(cid:117)(cid:116) 1 GG (cid:88) i = 1 (mF1 i  mF1) 2 (13) We also report the worst-group performance (mF1 W = min([mF1 1 , mF1 2 , . . . mF1 G )). In Table 4, we report the results of all our baselines on the four datasets introduced in this paper. We first observe that the results of linear classifiers trained with the ERM algorithm (top row per dataset) are consistently worse (lower average and worst-case performance, higher group disparity) compared to transformed-based models in the same setting. In other words linear classifier have lower overall performance, while being less fair with respect to the applied definition of fairness (i.e. equal performance across groups). As one can see, transformer-based models trained with the ERM algorithm, i.e., without taking into account information about groups and their distribution, perform either better on in the same ballpark than models trained with methods specialized to mitigate biases (Section 4), with an average loss of 0 . 17% only in terms of mF 1 and of 0 . 78% in terms of mF 1 W . While, these algorithms improve worst case performance in the literature, 4396 ECtHR (ECHR Violation Prediction) SCOTUS (Issue Area Classification) Algorithm Defendant State Applicant Gender Applicant Age Respondent Type Direction  mF1  GD  mF1 W  mF1  GD  mF1 W  mF1  GD  mF1 W  mF1  GD  mF1 W  mF1  GD  mF1 WB ag of -W ords L inear C lassifier ERM 46.8 3.0 43.8 44.1 4.9 40.6 46.9 6.3 40.9 73.8 6.6 61.8 77.5 2.6 74.9 T ransformer based C lassifier ERM 53.2 8.3 44.9 57.5 3.1 54.4 54.1 5.9 46.2 75.1 4.0 70.8 78.1 1.6 76.6 ERM + GS 54.4 5.5 48.9 57.8 3.3 54.5 56.0 5.6 48.7 75.2 3.9 70.9 77.1 1.3 76.0 ADV-R 53.8 5.8 47.9 54.6 3.2 51.5 48.9 6.1 40.6 56.9 4.7 53.1 41.0 0.8 40.3 G-DRO 55.0 5.2 49.8 56.3 1.9 55.0 52.6 6.2 44.3 74.5 3.3 71.6 77.1 1.7 75.4 IRM 53.8 5.7 48.1 53.8 2.3 52.5 54.8 4.4 49.5 73.4 4.8 68.2 78.1 2.7 75.4 V-REx 54.6 6.3 48.3 54.6 2.0 53.2 55.0 4.5 49.8 73.8 3.8 68.2 78.2 1.1 77.1 FSCS (Case Approval Prediction) CAIL (Crime Severity Prediction) Algorithm Language Legal Area Region Defendant Gender Region  mF1  GD  mF1 W  mF1  GD  mF1 W  mF1  GD  mF1 W  mF1  GD  mF1 W  mF1  GD  mF1 WB ag of -W ords L inear C lassifier ERM 55.5 6.2 46.8 54.4 9.7 40.9 56.8 5.0 46.6 33.5 0.7 32.8 31.7 5.0 25.5 T ransformer based C lassifier ERM 67.8 2.1 65.0 69.4 9.6 56.9 69.7 2.9 63.9 60.2 0.6 60.1 59.3 3.5 56.4 ERM + GS 66.4 3.5 61.7 67.1 9.3 55.5 67.9 3.0 62.3 59.4 0.7 59.1 58.2 3.1 55.9 ADV-R 62.6 5.1 59.0 65.6 12.4 50.0 67.4 3.2 61.5 53.3 1.3 52.1 53.5 2.5 50.8 G-DRO 70.5 0.6 69.9 57.5 5.6 52.6 67.7 4.2 60.2 59.2 1.3 57.9 58.9 3.7 55.7 IRM 68.3 1.9 66.7 67.8 9.5 55.8 68.7 3.0 63.2 56.4 1.5 55.7 58.0 3.1 54.9 V-REx 67.2 3.5 62.4 66.6 8.9 56.0 68.4 3.1 62.4 58.5 0.7 58.3 58.6 3.3 54.4 Table 4: Test results for all examined group-robust algorithms per dataset attribute. when applied in a controlled experimental environment, they fail in a more realistic setting, where both groups across attributes and labels are imbal-anced, while also both group and label distribution change over time. Furthermore, we cannot identify one algorithm that performs better across datasets and group with respect to the others, indeed results are quite mixed without any recognizable pattern. The current version of FairLex covers a very small fraction of legal applications, jurisdictions, and protected attributes. Our benchmark is open-ended and inevitably cannot cover  everything in the whole wide (legal) world  (Raji et al., 2021), but nonetheless we believe that the published resources will help critical research in the area of fairness. Some protected attributes within our datasets are extracted automatically, i.e., the gender and the age in the ECtHR dataset, if possible, by means of regular expressions, or manually clustered by the authors, such as the defendant state in the ECtHR dataset and the respondent attribute in the SCOTUS dataset. Various simplifications made, e.g, the binarization of gender, would be inappropriate in real-world applications. Another important limitation is that what is considered the ground truth in these datasets (with the exception of SCOTUS) is only ground truth relative to judges' interpretation of a specific (EC, US, Swiss, Chinese) jurisdiction and legal framework. The labeling is therefore somewhat subjective for non-trivial cases, and its validity is only relative to a given legal framework. We of course do not in any way endorse the legal standards or framework of the examined datasets. We introduced FairLex, a multi-lingual benchmark suite for the development and testing of models and bias-mitigation algorithms within the legal domain, based on four datasets covering four jurisdictions, five languages and various sensitive attributes. Furthermore, we provided competitive baselines including transformer-based language models adapted to the examined datasets, and examination of performance of four group robust algorithms (Adversarial Removal, IRM, Group DRO, and V-REx). While, these algorithms improve worst case performance in the literature, when applied in a controlled experimental environment, they fail in a more realistic setting, where both groups across attributes, and labels are imbal-anced, while also both group and label distributions change over time. Furthermore, we cannot identify a single algorithm that performs better across datasets and groups compared to the rest. In future work, we aim to further expand the benchmark with more datasets that could possibly cover more sensitive attributes. Further analysis on the reasons behind group disparities, e.g., representational bias, systemic bias, is also critical. The scope of this work is to provide an evaluation framework along with extensive experiments to further study fairness within the legal domain. Following the work of Angwin et al. (2016), Dressel and Farid (2018), and Wang et al. (2021b), we provide a diverse benchmark covering multiple tasks, jurisdictions, and protected (examined) attributes. We conduct experiments based on pre-trained transformer-based language models and compare model performance across four representative group-robust algorithm, i.e., Adversarial Removal (Elazar and Goldberg, 2018), Group DRO (Sagawa et al., 2020), IRM (Arjovsky et al., 2020) and REx (Krueger et al., 2020). We believe that this work can inform and help practitioners to build assisting technology for legal professionals with respect to the legal framework (jurisdiction) they operate -; technology that does not only rely on performance on majority groups, but also considering minorities and the robustness of the developed models across them. We believe that this is an important application field, where more research should be conducted (Tsarapatsa-nis and Aletras, 2021) in order to improve legal services and democratize law, but more importantly highlight (inform the audience on) the various multi-aspect shortcomings seeking a responsible and ethical (fair) deployment of technology. We standardize and put together four datasets: ECtHR (Chalkidis et al., 2021), SCOTUS (Spaeth et al., 2020), FSCS (Niklaus et al., 2021), and CAIL (Xiao et al., 2018; Wang et al., 2021b) that are already publicly available under CC-BY-(NC-)SA-4.0 licenses. We release the compiled version of the dataset under a CC-BY-NC-SA-4.0 license to favor academic research, and forbid to the best of our ability potential commercial dual use. 15 All datasets, except SCOTUS, are publicly available and have been previously published. If datasets or the papers where they were introduced in were not compiled or written by ourselves, we have referenced the original work and encourage FairLex users to do so as well. In fact, we believe that this work should only be referenced, in addition to citing the original work, when jointly experi-15 https://creativecommons.org/licenses/ by-nc-sa/4.0/ menting with multiple FairLex datasets and using the FairLex evaluation framework and infrastructure, or use any newly introduced annotations (EC-tHR, SCOTUS). Otherwise only the original work should be cited. The data is in general partially anonymized in accordance with the applicable national law. The data is considered to be in the public sphere from a privacy perspective. This is a very sensitive matter, as the courts try to keep a balance between transparency (the public's right to know) and privacy (respect for private and family life). ECtHR cases are partially annonymized by the court. Its data is processed and made public in accordance with the European data protection laws. SCOTUS cases may also contain personal information and the data is processed and made available by the US Supreme Court, whose proceedings are public. While this ensures compliance with US law, it is very likely that similarly to the ECtHR any processing could be justified by either implied consent or legitimate interest under European law. In FSCS, the names of the parties have been redacted by the courts according to the o  cial guidelines. CAIL cases are also partially anonymized by the courts according to the courts' policy. Its data is processed and made public in accordance with Chinese Law. This work is fully funded by the Innovation Fund Denmark (IFD) 16 under File No. 0175-00011A. We would like to thank the authors of the original datasets for providing access to the original documents, metadata, or confidentially sharing pre-released versions of the datasets.
method abstain result result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other method abstain abstain method abstain abstain abstain method method abstain objective objective objective method objective objective objective objective objective method result other other method method method other other method other other other other other other other other abstain method other other method other other other method objective abstain method abstain method abstain abstain abstain abstain method abstain other abstain abstain method abstain abstain method abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method other method method method method method method result result result result abstain result abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result abstain abstain result abstain abstain abstain abstain abstain other abstain result abstain method method abstain method abstain abstain abstain method abstain method objective abstain method method method abstain abstain abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other