title: Empowering Cross-lingual Behavioral Testing of NLP Models with
    Typological Features
summary: A challenge towards developing NLP systems for the world's languages is
  understanding how they generalize to typological differences relevant for
  real-world applications. To this end, we propose M2C, a morphologically-aware
  framework for behavioral testing of NLP models. We use M2C to generate tests
  that probe models' behavior in light of specific linguistic features in 12
  typologically diverse languages. We evaluate state-of-the-art language models
  on the generated tests. While models excel at most tests in English, we
  highlight generalization failures to specific typological characteristics such
  as temporal expressions in Swahili and compounding possessives in Finish. Our
  findings motivate the development of models that address these blind spots.
paper_authors._id: 64adfed880f308a395f3823e
paper_authors.name: Ester Hlavnova
paper_authors._id: 64adfed880f308a395f3823f
paper_authors.name: Sebastian Ruder
paper_title: Empowering Cross-lingual Behavioral Testing of NLP Models with
    Typological Features
paper_summary: A challenge towards developing NLP systems for the world's languages is
  understanding how they generalize to typological differences relevant for
  real-world applications. To this end, we propose M2C, a morphologically-aware
  framework for behavioral testing of NLP models. We use M2C to generate tests
  that probe models' behavior in light of specific linguistic features in 12
  typologically diverse languages. We evaluate state-of-the-art language models
  on the generated tests. While models excel at most tests in English, we
  highlight generalization failures to specific typological characteristics such
  as temporal expressions in Swahili and compounding possessives in Finish. Our
  findings motivate the development of models that address these blind spots.
paper_ai_summary: M2C, a morphologically-aware framework, tests and highlights generalization failures of NLP models to specific typological characteristics across 12 diverse languages.
paper_ai_keywords: morphologically-aware behavioral testing typological differences temporal expressions compounding possessives