---
abstract: 'The relation between language and thought has occupied linguists for at least a century. Neo-Whorfianism, a weak version of the controversial Sapir–Whorf hypothesis, holds that our thoughts are subtly influenced by the grammatical structures of our native language. One area of investigation in this vein focuses on how the grammatical gender of nouns affects the way we perceive the corresponding objects. For instance, does the fact that [*key*]{} is masculine in German ([*der Schl[ü]{}ssel*]{}), but feminine in Spanish ([*la llave*]{}) change the speakers’ views of those objects? Psycholinguistic evidence presented by suggested the answer might be yes: When asked to produce adjectives that best described a [*key*]{}, German and Spanish speakers named more stereotypically masculine and feminine ones, respectively. However, recent attempts to replicate those experiments have failed [@mickan2014key]. In this work, we offer a computational analogue of ’s experimental design on 9 languages, finding evidence *against* neo-Whorfianism.'
author:
- |
    Katharina Kann\
    New York University, USA\
    `kann@nyu.edu`
bibliography:
- 'neowhorf.bib'
title: 'Grammatical Gender, Neo-Whorfianism, and Word Embeddings: A Data-Driven Approach to Linguistic Relativity'
---

Introduction {#sec:introduction}
============

During his tenure as a graduate student, 20$^\text{th}$-century American linguist Benjamin Whorf conducted field work on Hopi, an Uto-Aztecan language spoken in Southern Arizona. To his surprise, he found that Hopi does not mark the tense of a verb in the way many Western European languages do [@whorf1956hopi].[^1] Thus, according to Whorf, a Hopi speaker must infer whether an action takes place in the past, present or future only from the sentential context in which the verb occurs. This finding inspired Whorf to start questioning whether language influences thought, a position that has come to be known as linguistic relativity [@whorf2012language]. Ultimately, Whorf went on to hypothesize that the Hopi perceive time differently as a result of their language’s grammar, kicking off a more encompassing debate on the relation of language and thought and engendering one of the larger controversies in linguistics to date [@deutscher2010through].

![image](figures/sentence.pdf){width="\textwidth"}

While influential, the strong version of what has come to be known as the Sapir–Whorf[^2] hypothesis has been disproven. For instance, even though languages differ in the basic color terms they employ, e.g., Korean has one word that represents both green and blue, the development of color terminology and perception is subject to universalist constraints due to biology [@berlin1969basic]. Recent years, however, have witnessed a resurgence of a milder strain of the hypothesis, alternatively known as neo-Whorfianism or the weak Sapir–Whorf hypothesis [@boroditsky2003linguistic]. One prominent controversial research direction investigates this hypothesis with respect to grammaticalized notions of gender and tense. Based on experimental evidence obtained from native German and Spanish speakers, argued that grammatical gender affects how speakers view objects in their native language. Similarly, differences in the usage of tense in Mandarin and English were found to affect how the respective speakers view time [@Boroditsky01doeslanguage]. In this work, we focus on the influence of grammatical gender and how we can use NLP tools to shed light on the validity of this aspect of neo-Whorfianism.

The gist of ’s hypothesis is that speakers of languages that mark grammatical gender perceive nouns, even inanimate ones, differently depending on the noun’s gender. They sought psycholinguistic evidence for this claim, arguing that speakers will more often choose stereotypically masculine adjectives to describe grammatically masculine inanimate nouns and stereotypically feminine adjectives to describe grammatically feminine inanimate nouns. To give a concrete example, the German word for key, [*Schl[ü]{}ssel*]{}, happens to be masculine, so speakers are more likely to use words such as [*heavy*]{}, [*jagged*]{}, and [*hard*]{}. In contrast, the Spanish word for key, [*llave*]{}, happens to be feminine, so speakers are more likely to use stereotypically feminine words such as [*elegant*]{}, [*pretty*]{}, and [*delicate*]{}. However, failed to replicate this experiment, leaving the validity of the findings questionable.

In this work, we provide evidence against ’s hypothesis using NLP techniques. We contend that if this instance of neo-Whorfianism is true, then we should see a reflection of it in corpus co-occurrence counts. We propose two experiments, based on word embeddings trained on large corpora, to investigate how speakers of different languages use certain nouns. Our results on 9 languages suggest that the grammatical gender of inanimate nouns does not influence how they are used in context, taking credence away from ’s claims. However, we caution that it is important not to overstate the findings of one study; we see our results as providing further evidence in the linguistic relativity debate against neo-Whorfianism from a largely orthogonal source.

Background {#sec:background}
==========

Grammatical Gender {#sec:grammatical-gender}
------------------

In this section, we provide a brief overview of grammatical gender systems since those play an important role in ’s, and hence our, experiments. Languages range from encoding no grammatical gender on inanimate nouns, like English or Mandarin Chinese, to distinguishing tens of gender-like noun classes, as found in the Bantu languages of Africa [@corbett1991gender]. In this work, we will exclusively consider gendered languages of Indo-European and Afro-Asiatic (Semitic) stock; they all distinguish either two or three genders: the bipartite distinctions masculine-feminine and the tripartite distinction masculine-feminine-neuter. An important assumption of our study is the tenet that the gender assigned to inanimate objects is arbitrary.

All languages we experiment on exhibit **concord** in grammatical gender, i.e., articles and adjectives that modify a noun must agree with that noun in gender as well as in other features. Consider the German and Spanish translations of the English sentence: *The beautiful key is on the table*. $$\begin{aligned}
  &\textit{La llave bonita est{\'a} sobre la mesa.} \\ &\textit{Der
    sch{\"on}e Schl{\"u}ssel liegt auf dem Tisch.}\end{aligned}$$ Here, German employs the masculine article [*der*]{} as [*Schl[ü]{}ssel*]{} is masculine; had [*Schl[ü]{}ssel*]{} been feminine, German speakers would say [*die Schl[ü]{}ssel*]{}. Spanish exhibits a similar behavior, making use of [*la*]{}, rather than [*el*]{}, as [*llave*]{} is feminine. For a thorough treatment of the subject, see .

A key part of our experimental design will be stripping side effects of the grammatical concord from articles, adjectives, and verbs, thus removing overt signals of the noun’s gender.

Morphological Tagging and Lemmatization {#sec:lemmatization}
---------------------------------------

Our study will make use of NLP tools that automate bits of linguistic analysis: **morphological tagging** and **lemmatization**. Both will be explained here.

In languages that exhibit inflectional morphology, we may decompose a word into a bundle of morpho-syntactic features and a lemma, its canonical form. Formally, we denote a word in a natural language as $w$, and a sentence of length $n$ as $\mathbf{w} = w_1 \cdots w_i \cdots w_n$. Each word can be factored into a lemma $\ell$ and a bundle of morphological features $m$. For instance, we may think of the German word [*Schl[ü]{}ssels*]{} as the lemma $\ell = {\textit{Schl{\"u}ssel}}$ and the features $m =
\left[\right.\textsc{pos}$$=$$\textit{n},\enskip \textsc{gen}$$=$$\textit{masc},\enskip \textsc{num}$$=$$\textit{sg}, \enskip\textsc{case}$$=$$\textit{gen}\left.\right]$. Each morphological feature in $m$ is an attribute–value pair. Attributes encompass lexical properties such as gender, number, and case, taking values such as masculine, singular and genitive, respectively. Following , we may divide the morphological attributes into two categories: **inherent** and **contextual**. Inherent categories are those that are embedded in the lemma itself: the lemma [*Schl[ü]{}ssel*]{} without any additional inflection reveals $\textsc{pos}$$=$$\textit{n}$ and $\textsc{gen}$$=$$\textit{masc}$. However, the sentential context dictates that $\textsc{num}$$=$$\textit{sg}$ and $\textsc{case}$$=$$\textit{gen}$. We write $\mathbf{m}$ for a sequence of morphological tags and ${\boldsymbol \ell}$ for a sequence of lemmata. A morphologically tagged Russian example sentence is given in \[fig:sentence\].

In general, each word in a sentential context will have exactly one lemma and one bundle of morpho-syntactic attributes. We may think of a decomposed sentence of length $n$ as an interleaved trisequence: $\langle w_1, \ell_1, m_1\rangle \cdots \langle w_n, \ell_n, m_n\rangle$, where $w_i$ is the $i^\text{th}$ word, $\ell_i$ is its lemma, and $m_i$ is the set of its morphological features. Several techniques exist to map sentences to the lemmata of their words together with their morpho-syntactic attributes. The task of mapping a sentence to a sequence of morphological tags, in above notation $\mathbf{w} \mapsto \mathbf{m}$, is known as morphological tagging. The task of mapping a sequence of words to a sequence of lemmata, i.e., $\mathbf{w} \mapsto {\boldsymbol \ell}$, is known as lemmatization. Performing the tasks jointly can improve performance [@muller-EtAl:2015:EMNLP].

[width=]{}

          bg     es     fr     he     it     pl     ro     ru     sk                                       
  ----- ------ ------ ------ ------ ------ ------ ------ ------ ------ -- -- -- -- -- -- -- -- -- -- -- -- --
   all   96.6   98.5   98.4   96.5   98.2   96.5   98.1   96.5   95.0                                      
   OOV   82.1   91.7   89.6   79.6   91.2   86.9   89.0   88.1   86.6                                      

  : Token-level lemmatization accuracy obtained by <span style="font-variant:small-caps;">lemming</span> on the test splits of the UD treebanks for all languages when evaluated on all tokens or just OOVs.[]{data-label="tab:lemming"}

Word Embeddings {#sec:word-embeddings}
---------------

In our experiments, we will make strong use of embeddings of words into $\mathbb{R}^d$. Given a fixed vocabulary $V = \{v_1, \ldots, v_{|V|}\}$ of word types, we will denote the embedding of a type $v$ as $\mathbf{e}(v) \in \mathbb{R}^d$.[^3]

We employ the <span style="font-variant:small-caps;">word2vec</span> [@mikolov2013efficient; @goldberg2014word2vec] toolkit, in particular the skip-gram model, for the creation of our word embeddings. Skip-gram may be considered a form of matrix factorization; specifically, it factorizes a matrix of probabilities $X \in \mathbb{R}^{|V| \times |V|}$, where $X_{ij}$ denotes the probability that $v_i$ co-occurs with $v_j$ within a certain context window. For instance, a symmetric context window of size 5 is a common choice. This asks how often $v_j$ occurs within five positions to the left or right of $v_i$. Under this interpretation, <span style="font-variant:small-caps;">word2vec</span> is an instance of exponential-family PCA [@DBLP:conf/nips/CollinsDS01; @cotterell-et-al-2017-eacl]. The output of <span style="font-variant:small-caps;">word2vec</span> is a mapping of word types to a vector space: $\mathbf{e} : V \rightarrow \mathbb{R}^d$. We highlight that the model only considers words categorically, i.e., it is unable to look at the surface form of the word. This is relevant as there are sub-word indicators of gender, e.g., feminine nouns in Spanish often end in [*-a*]{}.

In practice, embeddings are taken as a proxy for lexical semantic meaning—words that have similar meanings should be closer together in the space. This idea has a long history in NLP; famously quipped, “You shall know a word by the company it keeps.” As we will discuss in \[sec:original\], are we interested in how well the embeddings for *inanimate nouns* encode the respective genders.

Neo-Whorfianism and Word Embeddings: What is the Link? {#sec:original}
======================================================

Our primary contribution is the development of a computational analogue of the previously conducted psycholinguistic study by , mentioned in \[sec:introduction\]. While, naturally, the signals in a big-data analysis are different than those extracted from subjects in a laboratory, we find it useful to first describe the original work.

The Psycholinguistic Experiment {#sec:psycho}
-------------------------------

To test the hypothesis that the grammatical gender assigned to inanimate objects, even though it is not (and, in fact, *cannot* be) a direct reflection of natural gender (since the latter is not defined for inanimate nouns), has an influence on the manner in which speakers perceive those objects, use the following experimental procedure. They created a list of 24 words in German and Spanish that were selected to be translations of each other. Importantly, the same number of masculine and feminine nouns were present in each language; however, all nouns had different genders in German and Spanish. We show a list of stimuli for the experiment in \[tab:stimuli\], taken from . Then, native speakers of each language were brought into the laboratory and asked to describe the nouns with the first three adjectives that came to mind. (Note the that the experiment took place in English, even though each subject’s native language was either German or Spanish.) As a second step, a group of native English speakers were asked to judge the elicited adjectives as either $-1$ (feminine) or $+1$ (masculine), which yielded a gender rating. Using this rating, found a correlation between the genderedness of the German and Spanish speakers’ choice of adjectives in the first experiment and the English speakers’ rating of how gendered each adjective was. This was taken to be a very conservative test for neo-Whorfian effects regarding gender, as the experiment took place in English, which lacks grammatical gender.

As a qualitative example, they report that the German-speaking participants described “key,” which is masculine in German, as [*hard*]{}, [*heavy*]{}, and [*jagged*]{}, whereas the Spanish-speaking participants described “key,” which is feminine in Spanish, as [*beautiful*]{}, [*elegant*]{}, and [*fragile*]{}. They argue that these findings show that the manner in which German and Spanish speakers think about inanimate objects is influenced by the grammatical gender that the language assigns to their nouns.

The findings of have not gone uncontested. report two unsuccessful attemps to replicate the experiments described in \[sec:original\]. They also note that, while the study is widely cited, the experiments, along with their experimental stimuli, were never published in their own right, but rather merely described in a summary book chapter. Nevertheless, the idea that grammatical gender may influence thought has taken off with much ink spilled on the subject in the popular press. Indeed, in partial response to the popularity of the idea, authored an entire volume, *The Language Hoax*, with the explicit purpose of removing much of the hype surrounding neo-Whorfian claims.

Neo-Whorfianism, Gender, and Word Embeddings {#sec:neo-whorphianism-gender-embeddings}
--------------------------------------------

How can we use NLP to test the claims investigated in \[sec:psycho\]? We contend that the thrust of ’s argument may be reduced to one of basic lemma co-occurrence counts in a large corpus. If German speakers are more likely to describe a [*Schl[ü]{}ssel*]{} ([*key*]{}) with a stereotypically masculine adjective, i.e., [*jagged*]{}, rather than with a stereotypically feminine one, i.e., [*delicate*]{}, we should reasonably expect this predisposition to manifest itself in corpus counts: grammatically masculine nouns should have stereotypically masculine adjectives modifying them with higher frequency, while feminine nouns should be more likely to be modified by stereotypically feminine ones.

Linking this idea to NLP, recall from \[sec:word-embeddings\] that co-occurrence counts are the primary signal for training word embeddings, one of the most popular lexical semantic representations of meaning at the type level. Expanding on ’s original statement, we further ask if you shall also know the word’s *grammatical gender* from its kept company? Operationalizing this, we will attempt to predict the gender of a noun type from its word embedding under several experimental conditions. This is a reformulation of ’s experimental paradigm. The original experiment asked the participants to generate adjectives given a noun stimulus, whereas we look at the spontaneously written contexts of nouns in large corpora. In more detail, ’s participants are given nouns as stimuli, whose gender they assume the participants have access to in their internal representation of the lexicon. (Recall that gender is an inherent morphological property, as discussed in \[sec:grammatical-gender\].) Then, those participants are asked to generate adjectives that would be good descriptors for those nouns, i.e., they generate contexts for those nouns. The contexts are then scored in a second experiment, where English speakers determined how gender-stereotyped the adjectives are. In our experiments, we induce a representation of the noun’s context from copious amounts of raw text. (The context will include adjectives, like the ones ’s participants generate.) Then, we employ a classifier to reconstruct the gender of the inanimate noun from its embedded (lemmatized) context. Despite this difference, the idea tested is identical: can one predict a noun’s grammatical gender from the words humans associate with it?

Methodologically, we will introduce two experiments that investigate the relation between a noun’s grammatical gender and the noun’s context. Experiment 1 (\[sec:experiment1\]) attempts to predict the gender from a word embedding with a black-box classifier, as we hint at above. Experiment 2 (\[sec:experiment2\]) complements experiment 1 in that it provides a technique for analyzing the “genderedness” of inanimate nouns. Both experiments make used of the same multilingual data, whose preparation is described in \[sec:preparation\].

Experimental Setup {#sec:preparation}
==================

The goal of both our experiment and that of is to determine whether the words that occur in the context of a noun are influenced by its grammatical gender. In our experiment, we opt to represent a context by its word embedding and try to predict the gender of a (inanimate) noun given its word embedding.

Extracting the gender of a word from its vector representation is often trivial for many languages due to certain grammatical artifacts: e.g., in Spanish, nouns are usually accompanied by a gender-specific article ([*el*]{} or [*la*]{}). Thus, in order to obtain meaningful experimental results, we need to control for such obvious indicators, i.e., lemmatize our corpora and train lemma embeddings instead of embeddings for all inflected forms. Indeed, word embeddings famously capture gender, as evinced by ’s (approximate) equation $$\mathbf{e}(\text{{\textit{king}}}) - \mathbf{e}(\text{{\textit{man}}}) + \mathbf{e}(\text{{\textit{woman}}}) \approx \mathbf{e}(\text{{\textit{queen}}}).$$ Recall from \[sec:word-embeddings\] that our embeddings do not have access to subword information, so clues for a noun’s gender *must* come from context.

Word Embedding Comparison
-------------------------

We induce word embeddings under four experimental conditions: (i) **forms**: the embeddings are trained on forms (the original corpus), (ii) **lemmata**: the embeddings are trained on lemmata (the whole corpus is lemmatized), (iii) **nouns**: the embeddings are trained on lemmatized nouns where the rest of the corpus is left unlemmatized, (iv) **$\neg$nouns**: the embeddings are trained on unlemmatized nouns and the rest are lemmatized.

#### Hypotheses.

We compare the ability of the classifier to predict the gender of the noun in the word conditions outlined above and, additionally, compare the results to a majority-class baseline. We hypothesize the classifier to easily be able to predict the gender from conditions (i) **forms** and (iii) **nouns** since the primary cue for gender is the concord exhibited by the context words. Indeed, we see no *a-priori* reason why (i) should perform significantly differently than (iii). The conditions (ii) **lemmata** and (iv) **$\neg$nouns** are more interesting: If the inherent gender in the inanimate nouns influences the choice of context lemmata, as believe, then we hypothesize the classifier to be able to predict the gender from the embeddings in (ii) **lemmata** and (iv) **$\neg$nouns** better than a majority-class baseline. However, if speakers are uninfluenced by grammatical gender, then we should fail to predict grammatical gender from context. We note (i) and (iii) are skylines since (ii) and (iv) contain less gender-related information.

Data Preparation {#sec:preparation}
----------------

The first step in the data creation pipeline is lemmatization. This is necessary to remove grammatical concord, for instance, because the Spanish phrase [*la llave bonita*]{} shows concord between the feminine noun [*llave*]{}, the definite article [*la*]{} and the adjective [*bonita*]{}. The lemmatized version of the sentence is [*el llave bonito*]{}, where both the article and the adjective have been reverted to their canonical form, which, traditionally, is the masculine singular form in Spanish. Hence, gender information is no longer trivially detectable using neighboring words.

#### Lemmatizer.

We choose the <span style="font-variant:small-caps;">lemming</span> [@muller-EtAl:2015:EMNLP] package[^4] for lemmatization. <span style="font-variant:small-caps;">lemming</span> is a conditional random field [@lafferty2001conditional] with artisanal feature templates. While occasionally surpassed in performance, <span style="font-variant:small-caps;">lemming</span> is competitive with neural state-of-the-art models, while remaining robust in low-resource settings and being fast to train [@heigold-neumann-vangenabith:2017:EACLlong]. For all languages in our experiments, we keep <span style="font-variant:small-caps;">lemming</span>’s defaults as our hyperparameters and train it on the training and development splits of the corresponding Universal Dependencies (UD) treebanks [@11234/1-1983]. Since errors during lemmatization could significantly alter our experiments, we make use of the dev splits to ensure that a high lemmatization performance is reached. \[tab:lemming\] shows the resulting token-based accuracies of our final models, both on the entire UD test sets and on out-of-vocabulary words only. The latter serves as a lower-bound; it corresponds to the (extreme) case where none of the words of the corresponding Wikipedia corpus appear in the union of the UD training and development sets.

#### Experimental Languages.

We choose 9 experimental languages randomly from UD: Bulgarian (bg), French (fr), Hebrew (he), Italian (it), Polish (pl), Romanian (ro), Russian (ru), Slovak (sk), Spanish (es). The only imposed condition is that that they have $\geq 80\%$ performance on lemmatizing out-of-vocabulary words on the UD development set. This is to ensure that our lemmatizer is reasonably leak-free for rare words. We drop neuter words in languages that exhibit a neuter, such as Bulgarian and Russian.

#### Word Vectors.

We employ the skip-gram model from the <span style="font-variant:small-caps;">word2vec</span> package to induce 100-dimensional embeddings. We use negative sampling with 10 samples. For all languages, the vectors are trained on corpora lemmatized in the way we just described; namely, we make use of multilingual Wikipedia editions from March 2018. All words with a frequency below $5$ are ignored, and we compare symmetric context window sizes of $2$, $5$ and $10$, finding $2$ works the best.

Experiment 1: Gender Classification {#sec:experiment1}
-----------------------------------

Our gender prediction problem constitutes a binary classification task, where the classes are masculine and feminine and the input is the word embedding of a noun. We employ a multi-layer perceptron (MLP) for this classification, defining the probability of the gender $g$ given a noun type $v$, as $$\begin{aligned}
\label{eq:mlp}
&p(g \mid v) =\notag\\
&\quad\text{softmax}(W_2 \tanh \left(W_1 \mathbf{e}(v) +
b_1\right) + b_2) \end{aligned}$$ where we feed in the noun’s embedding $\mathbf{e}(v)$ into the network, $W_1 \in
\mathbb{R}^{d' \times d}$ and $W_2 \in \mathbb{R}^{2 \times d'}$ are weight matrices, $b_1 \in \mathbb{R}^{d'}$ and $b_2 \in
\mathbb{R}^{2}$ are bias vectors. represents a network of depth 2, but we consider depth-$k$ networks where $k$ is a hyperparameter. We additionally consider the non-linearities $\textit{ReLU}$ and $\textit{sigmoid}$.

![Accuracies for our classifer on all languages. The majority-class baseline (blue bar) is generally above the lemmata-based prediction (green bar), but much below the form-based prediction (yellow bar). This leads us to conclude that we *cannot* reliably predict grammatical gender for inanimate nouns from context alone.[]{data-label="fig:gender_results"}](graphs/gender_new){width="\columnwidth"}

#### Training Set.

To learn the parameters of \[eq:mlp\], we construct the following training set. Given the lemmatized and morphologically analyzed Wikipedias discussed in \[sec:preparation\], we construct a lexicon as follows. For every lemma type that occurs more than 50 times, we find the gender, extracted from the morphological tag of each token, that most frequently occurs among its tokens in the corpus. This yields a lexicon of lemma-gender pairs. Note that this training set will include animate words as we are unable to exclude them easily. However, we will only evaluate on inanimate nouns; see below.

#### Evaluation Set.

For the evaluation, we focus exclusively on the same common, inanimate words in all languages. We use the NorthEuraLex dataset [@northeuralex],[^5] which is a multi-way concept-aligned dictionary. In order to avoid biological gender interfering with grammatical gender and, thus, influencing our experiments, we manually annotate all concepts as animate or inanimate and exclude all animate nouns. For instance, we keep *eye*, *lake*, and *circle*, while discarding words like *wife*, *dog*, or *son*. The list containing all concepts in our evaluation set can be found in \[sec:northeuralex\]. We take care to remove these words from the training set. Note that not all words are observed in each language, due to the varying sizes of the Wikipedia corpora. We split the resulting lexicons in half, creating a dev and test set for each language.

#### Training and Hyperparameters.

The model is implemented in PyTorch [@paszke2017automatic]. We train our models on the training sets using Adam [@KinBa17] with a base learning rate of $0.1$ for all models. All models are trained for $50$ epochs. Hyperparameters include network depth ($k \in [1, 5]$), size of the hidden layer (taken from $\{100, 200, 300\}$) and type of nonlinearity (taken from $\{\tanh, \textit{sigmoid}, \textit{ReLU} \}$). We randomly partition the evaluation set in half 10 times and sweep the hyperparameters on each dev partition, performing early stopping. Final results average the performance on test across these splits.

[width=]{}

  ------------------------------------------- ------------------- -------------------------- ----------------------
                                                                                             
  (l)[1-2]{} (l)[3-4]{} **fem**               **masc**            **fem**                    **masc**
  (l)[1-2]{} (l)[3-4]{} [*estrella*]{} (f.)   [*sonido*]{} (m.)   [*labio*]{} (m.)           [*hoguera*]{} (f.)
  [*ola*]{} (f.)                              [*labio*]{} (m.)    [*mejilla*]{} (f.)         [*aldea*]{} (f.)
  [*bolsa*]{} (f.)                            [*tejido*]{} (m.)   [*entendimiento*]{} (m.)   [*esqu[í]{}*]{} (m.)
  [*isla*]{} (f.)                             [*lazo*]{} (m.)     [*[á]{}ngulo*]{} (m.)      [*altitud*]{} (f.)
  [*carta*]{} (f.)                            [*nudo*]{} (m.)     [*vena*]{} (f.)            [*silla*]{} (f.)
  ------------------------------------------- ------------------- -------------------------- ----------------------

  : The most gendered nouns in our Spanish test set according to our induced gender dimension under the **forms** and the **lemmata** conditions. The gender dimension is the column header and the true gender is marked in parentheses. \[tab:most\_gender\]

#### Results and Discussion.

The gender-prediction accuracies from the four embeddings conditions are shown in \[fig:gender\_results\]. As discussed, classification accuracy is highest for conditions (i) **forms** and (iii) **nouns**, i.e., embeddings trained on the corpora where the context words are unaltered. In these conditions, where we see unlemmatized forms as context, our classifier handily surpasses the majority-class baseline with differences up to 30 points. All differences are significant ($p < 0.05$). On the other hand, inspecting conditions (ii) **lemmata** and (iv) $\neg$**nouns**, we see that performance of each is rarely better than the majority-class baseline and in *no case* is statistically better ($p < 0.05$). Thus, despite a relatively extensive hyperparameter search, we are *unable* to reliably predict grammatical gender from the context words along. This negative result provides evidence *against* ’s hypothesis that the inherent gender of the word will have an effect on the context words that a speaker uses for inanimate nouns.

Experiment 2: A Gender Dimension {#sec:experiment2}
--------------------------------

In addition to Experiment 1, we would also like to analyze the degree to which words are more masculine or feminine using their word embeddings, which, in turn, tells us how masculine or feminine their contexts are. A high-level overview of how we can achieve this is as follows. We may isolate genderedness by fixing one of the dimensions of the word embeddings to be the gender dimension. Then, we seek a method that will shift the information regarding gender into that dimension. In effect, we need a method which maps every word to a scalar quantity which corresponds to how gendered the word is, allowing us to compare individual words and to discover those whose gender is more saliently encoded in the embeddings. We adopt the *ultra-dense* strategy developed by , which we will describe below.

#### Learning Ultra-Dense Embeddings.

Given an embedding $\mathbf{e}(v) \in \mathbb{R}^d$ of a word $v$, we are interested in learning a real orthogonal matrix $Q \in \mathbb{R}^{d \times d}$ in order to create a new embedding $\mathbf{e}'(v) = Q\, \mathbf{e}(v)$. Defining $Q$ to be real orthogonal ensures that no information is lost or gained as a result of the transformation—the dot product, and, thus, the cosine similarity between vectors will be preserved. In order to learn a transformation that moves gender information to certain components of the embeddings, let ${\cal S}$ be the set of of all pairs of distinct nouns that have the *same* gender and let ${\cal D}$ be the set of all pairs of distinct nouns that have a different gender. Let $P \in \mathbb{R}^{d \times d}$ be a matrix with all entries being zero except for $P_{11}$, which is 1. Now, we minimize the following objective $$\begin{aligned}
  \mathcal{O}\Big(Q; \mathcal{S}, &\mathcal{D}\Big) = \!\!\!\sum_{(v, v') \in {\cal S}} \!\!\! \mid\mid P\, Q\left(\mathbf{e}(v) - \mathbf{e}(v')\right)\mid\mid^2_2 \nonumber  \\
     -\!\!\!&\sum_{(v, v') \in {\cal D}} \!\!\! \mid\mid P\, Q\left(\mathbf{e}(v) - \mathbf{e}(v')\right)\mid\mid^2_2  \label{eq:obj_fct}\end{aligned}$$ with respect to the matrix $Q$ subject to the constraint that $Q^{\top} Q = I$; that is, $Q$ is real orthogonal.

#### Stochastic Projected Gradient.

The above objective can be optimized using a stochastic projected-gradient-style algorithm [@bertsekas1999nonlinear]. This algorithm alternates between two steps until convergence: (i) A stochastic gradient step: During this step, one element is randomly sampled from each of ${\cal S}$ and ${\cal D}$. Then, the gradient of \[eq:obj\_fct\] is computed with respect to $Q$, and $Q$ is updated by taking a step in the direction of the gradient. (ii) A projection step: After obtaining a new matrix $Q' = Q + \eta \cdot \nabla_Q {\cal O}$ during the gradient update where $\eta$ is the learning rate, we no longer have the guarantee that $Q'$ is orthogonal. Thus, we must perform a projection step to orthogonalize $Q'$. This can be achieved through singular value decomposition (SVD). We compute the SVD: $Q' = U \Sigma V^{\top}$, where $U$, $\Sigma$ and $V^{\top}$ are guaranteed to be real as $Q'$ is real. Then, we may define $Q = UV^{\top}$, which is the closest to $Q'$ under the Frobenius norm [@horn1990matrix]. Pseudocode for this algorithm is given in \[alg:spga\].

$(v_{\cal S}, v_{\cal S}') \sim \text{uniform}({\cal S})$ $(v_{\cal D}, v_{\cal D}') \sim \text{uniform}({\cal D})$ $\tilde{{\cal S}} \gets {\left\{ (v_{\cal S}, v_{\cal S}') \right\}}; \tilde{{\cal D}} \gets {\left\{ (v_{\cal D}, v_{\cal D}') \right\}}$ $U\Sigma V^{\top} \gets \texttt{SVD}\left(Q'\right)$ $Q  \gets UV^{\top}$

#### Training Details.

Our learning rate schedule $\eta_t$ (see \[alg:spga\]) is chosen by the Adam optimizer [@KinBa17]. We run $T=1000$ iterations. Upon termination, we extract a scalar-valued gender quantity as follows: $\left[P\, Q\, \mathbf{e}(w)\right]_{11}$, i.e., the first component of the new embedding. We train on half the NorthEuraLex data and test on the other half, using the splits described in \[sec:experiment1\].

#### Analyzing the Data.

The gender dimension admits both a quantitative and a qualitative analysis. We start with a quantitative analysis; we consider Spearman’s $\rho$ between the gender dimension and the grammatical gender of the nouns, marking masculine as 0 and feminine as 1. The results are shown in \[table:corr\]. They mirror those found in experiment 1 (\[sec:experiment1\]): we are unable to find correlation significantly different than 0 for any of the cases where the context words have been lemmatized (conditions (i) **lemmata** and (iii) $\neg$**nouns**). On the contrary, when the context words are left unlemmatized (conditions (i) **forms** and (iii) **nouns**), we are generally able to find a significant correlation. Qualitatively, the gender dimension tells us which words are more gendered than others. Here, we perform a case study of our Spanish test set. The five words with the most masculine and most feminine gender dimension are displayed in \[tab:most\_gender\] in the (i) **forms** and (ii) **lemmata** conditions. The qualitative analysis shows the same trend.

  ----------------------- -- -- -- --
       [**Lang**]{}                
   (l)[1-1]{} (l)[2-5]{}           
            bg                     
            es                     
            fr                     
            he                     
            it                     
            pl                     
            ro                     
            ru                     
            sk                     
  ----------------------- -- -- -- --

  : \[table:corr\] Spearman’s $\rho$ between the generated dimension on test data and the true gender annotation. The correlation is statistically different than zero with $p < 0.05$ in those entries in blue; the ones in red were not found to be significant.

Other Related Work
==================

In the realm of NLP, the closest work to ours deals with bias in word embeddings. Many have observed that word embeddings encode the biases present in the data they were trained on on. For instance, and note that the [*engineer*]{} embedding has a higher cosine similarity with [*man*]{} than with [*woman*]{}, reflecting a structural imbalance in the gender of the profession; they propose to debias the embeddings such that gender no longer plays a role.

Conclusion
==========

Using word embeddings in 9 different languages trained on lemmatized corpora, we investigated whether adjective choice is influenced by the grammatical gender of inanimate nouns. This question has larger implication in the debate on the relation between language and thought. We developed a computational analogue of ’s experimental paradigm and showed that context in which a noun occurs, stripped of its overt gender markings, is no longer predictive of the inherent gender of the original, inanimate noun. These negative results contradict ’s claims.

Any scientific study should be viewed with a healthy dose of skepticism, especially one, such as ours, that considers a controversial question. We believe our big-data study should be taken as complementary evidence in the context of the larger debate that inanimate nouns’ gender does not influence the way speakers describe them in a corpus.

Acknowledgments {#acknowledgments .unnumbered}
===============

I would like to thank Hanna Wallach, Lawrence Wolf-Sonkin, and Ryan Cotterell for discussions and contributions, and they approve this acknowledgment. Except for the addition of author information, these acknowledgments, and minor changes this paper is unchanged from a manuscript written in 2018.

(Inanimate) NorthEuraLex Concepts {#sec:northeuralex}
=================================

[lr]{} Abend::N & evening\
Abhang::N & slope\
Abstand::N & gap\
Ader::N & vein\
Alter::N & age\
Angelegenheit::N & matter\
Anh[ö]{}he::N & elevation\
Anzahl::N & count\
Apfel::N & apple\
Arbeit::N & work\
Arm::N & arm\
Art::N & sort\
Arznei::N & medicine\
Asche::N & ashes\
Ast::N & limb\
Atem::N & breath\
Auge::N & eye\
Bach::N & brook\
Band::N & ribbon\
Bart::N & beard\
Bau::N & lair\
Bauch::N & belly\
Baum::N & tree\
Beere::N & berry\
Bein::N & leg\
Berg::N & mountain\
Besen::N & broom\
Bett::N & bed\
Beutel::N & pouch\
Bild::N & picture\
Birke::N & birch\
Blatt::N & leaf\
Blume::N & flower\
Blut::N & blood\
Boden::N & ground, soil\
Bogen[\[]{}Waffe[\]]{}::N & bow\
Boot::N & boat\
Brei::N & mush\
Brett::N & board\
Brief::N & letter\
Brot::N & bread\
Brunnen::N & well\
Brust::N & breast, chest\
Br[ü]{}cke::N & bridge\
Buch::N & book\
Buchstabe::N & character\
Bucht::N & cove\
Busen::N & bosom\
Butter::N & butter\
B[ü]{}ndel::N & bundle\
Dach::N & roof\
Decke::N & blanket\
Deckel::N & cover\
Donner::N & thunder\
Dorf::N & village\
Dreck::N & filth\
Ecke::N & corner\
Ei::N & egg\
Eimer::N & bucket\
Eis::N & ice\
Eisen::N & iron\
Ellenbogen::N & elbow\
Ende::N & end\
Entfernung::N & distance\
Erde::N & earth\
Erz[ä]{}hlung::N & story\
Essen::N & meal\
Faden::N & thread\
Falle::N & trap\
Farbe::N & paint\
Feder::N & feather\
Fehler::N & mistake\
Fell::N & fur\
Fenster::N & window\
Ferse::N & heel\
Festland::N & land\
Fett::N & fat\
Feuer::N & fire\
Fieber::N & fever\
Figur::N & figure\
Finger::N & finger\
Fingernagel::N & fingernail\
Fleisch::N & meat\
Fluss::N & river\
Fl[ü]{}gel::N & wing\
Frost::N & frost\
Funke::N & spark\
Fu[ß]{}::N & foot\
Fu[ß]{}boden::N & floor\
Gabel::N & fork\
Gang::N & walk\
Gast::N & guest\
Gedanke::N & thought\
Ged[ä]{}chtnis::N & memory\
Gegend::N & area\
Gegenstand::N & item\
Gehirn::N & brain\
Geist::N & spirit\
Geld::N & money\
Gel[ä]{}chter::N & laughter\
Genick::N & nape\
Geruch::N & odour\
Geschenk::N & gift\
Geschirr::N & dishes\
Geschmack::N & flavour\
Gesch[ä]{}ft::N & business\
Gesicht::N & face\
Gespr[ä]{}ch::N & talk\
Gesundheit::N & health\
Getreide::N & corn\
Gewalt::N & violence\
Gewehr::N & gun\
Gewicht::N & weight\
Gipfel::N & summit\
Glas::N & glass\
Gl[ü]{}ck::N & happiness\
Gold::N & gold\
Grab::N & grave\
Gras::N & grass\
Grenze::N & border\
Griff::N & handle\
Grube::N & pit\
Grund::N & reason\
Gr[ö]{}[ß]{}e::N & size\
G[ü]{}rtel::N & belt\
Haar::N & hair\
Haken::N & hook\
Hals::N & neck\
Hand::N & hand\
Handfl[ä]{}che::N & palm\
Handtuch::N & towel\
Haufen::N & heap\
Haus::N & house\
Haut::N & skin\
Heim::N & home\
Hemd::N & shirt\
Herz::N & heart\
Heu::N & hay\
Hilfe::N & help\
Himmel::N & sky\
Hitze::N & heat\
Holz::N & wood\
Honig::N & honey\
Horn::N & horn\
Hose::N & trousers\
Hunger::N & hunger\
H[ä]{}lfte::N & half\
H[ö]{}he::N & height\
H[ö]{}hle::N & cave\
H[ü]{}gel::N & hill\
Insel::N & island\
Jahr::N & year\
Kamm::N & comb\
Kampf::N & fight\
Kante::N & edge\
Kehle::N & throat\
Kessel::N & kettle\
Kiefer[\[]{}Anatomie[\]]{}::N & jaw\
Kiefer[\[]{}Baum[\]]{}::N & pine\
Kinn::N & chin\
Kirche::N & church\
Kissen::N & pillow\
Kiste::N & box\
Klaue::N & claw\
Kleidung::N & clothes\
Knie::N & knee\
Knochen::N & bone\
Knopf::N & button\
Knoten::N & knot\
Kohle::N & coal\
Kopf::N & head\
Korn::N & grain\
Kraft::N & force\
Kragen::N & collar\
Kralle::N & claw\
Krankheit::N & illness\
Kreis::N & circle\
Kreuz::N & cross\
Krieg::N & war\
Kummer::N & grief\
K[ä]{}lte::N & chill\
K[ö]{}rper::N & body\
K[ü]{}ste::N & coast\
Laden::N & shop\
Lagerfeuer::N & campfire\
Land::N & country\
Last::N & load\
Laut::N & sound\
Leben::N & life\
Leber::N & liver\
Leder::N & leather\
Lehm::N & clay\
Leine::N & leash\
Leiter::N & ladder\
Leute::N & people\
Licht::N & light\
Lied::N & song\
Linie::N & line\
Lippe::N & lip\
Loch::N & hole\
Luft::N & air\
Lust::N & desire\
L[ä]{}nge::N & length\
L[ä]{}rm::N & noise\
L[ö]{}ffel::N & spoon\
L[ü]{}ge::N & lie\
Macht::N & power\
Magen::N & stomach\
Meer::N & sea\
Menge::N & amount\
Messer::N & knife\
Milch::N & milk\
Mittag::N & noon\
Mitte::N & middle\
Monat::N & month\
Mond::N & moon\
Moor::N & moor\
Morgen::N & morning\
Mund::N & mouth\
Muster::N & pattern\
M[ä]{}rchen::N & fairy tale\
M[ü]{}tze::N & cap\
Nabel::N & navel\
Nachricht::N & message\
Nacht::N & night\
Nadel::N & needle\
Nagel::N & nail\
Nagel[\[]{}Anatomie[\]]{}::N & nail\
Nahrung::N & food\
Name::N & name\
Nase::N & nose\
Nebel::N & fog\
Nest::N & nest\
Netz::N & net\
Neuigkeit::N & news\
Norden::N & north\
Nutzen::N & profit\
Oberschenkel::N & thigh\
Ofen::N & stove\
Ohr::N & ear\
Ort::N & place\
Osten::N & east\
Pfad::N & path\
Pfeil::N & arrow\
Pilz::N & mushroom\
Platte::N & slab\
Platz::N & space\
Preis::N & price\
Puppe::N & doll\
Quelle::N & source\
Rand::N & fringe\
Rauch::N & smoke\
Raureif::N & hoarfrost\
Rede::N & speech\
Regal::N & shelf\
Regen::N & rain\
Regenbogen::N & rainbow\
Reichtum::N & wealth\
Reihe::N & row\
Riemen::N & strap\
Rinde::N & bark\
Ring::N & ring\
Rohr::N & pipe\
Ruder::N & oar\
Ruf::N & call\
Ruhe::N & calm\
R[ä]{}tsel::N & puzzle\
R[ü]{}cken::N & back, spine\
Saat::N & seed\
Sache::N & thing\
Sack::N & sack\
Salz::N & salt\
Sand::N & sand\
Schaden::N & damage\
Schale::N & husk\
Schatten::N & shadow\
Schaufel::N & shovel\
Schaum::N & foam\
Scheibe::N & slice\
Schlaf::N & sleep\
Schlinge::N & noose\
Schlitten::N & sleigh\
Schloss::N & lock\
Schluss::N & conclusion\
Schmerz::N & pain\
Schmutz::N & dirt\
Schnee::N & snow\
Schnur::N & string\
Schnurrbart::N & moustache\
Schritt::N & step\
Schuh::N & shoe\
Schuld::N & fault\
Schulter::N & shoulder\
Schwanz::N & tail\
See::N & lake\
Sehne::N & sinew\
Seite::N & side\
Silber::N & silver\
Sinn::N & meaning\
Ski::N & ski\
Sonne::N & sun\
Spaten::N & spade\
Speise::N & dish\
Spiegel::N & mirror\
Spiel::N & game\
Spitze::N & tip\
Sprache::N & language\
Spur::N & track\
Staat::N & state\
Stab::N & staff\
Stadt::N & town\
Stamm::N & trunk\
Stange::N & pole\
Staub::N & dust\
Stein::N & stone\
Stern::N & star\
Stiefel::N & boot\
Stimme::N & voice\
Stirn::N & forehead\
Stock::N & stick\
Stoff::N & cloth\
Stra[ß]{}e::N & road\
Strich::N & stroke\
Str[ö]{}mung::N & current\
Stuhl::N & chair\
St[ä]{}rke::N & strength\
St[ü]{}ck::N & piece\
St[ü]{}tze::N & bracket\
Sumpf::N & swamp\
Suppe::N & soup\
S[ü]{}den::N & south\
S[ü]{}nde::N & sin\
Tag::N & day\
Tanne::N & fir\
Tasche::N & bag\
Tasse::N & cup\
Tee::N & tea\
Teil::N & part\
Tisch::N & table\
Tod::N & death\
Ton::N & tone\
Topf::N & pot\
Tor::N & gate\
Traum::N & dream\
Tropfen::N & drop\
Tr[ä]{}ne::N & tear\
Tuch::N & scarf\
T[ü]{}r::N & door\
Ufer::N & shore\
Ungl[ü]{}ck::N & misfortune\
Verstand::N & mind\
Volk::N & nation\
Wahrheit::N & truth\
Wald::N & forest\
Wange::N & cheek\
Ware::N & ware\
Wasser::N & water\
Weg::N & way\
Weide::N & pasture\
Weide[\[]{}Baum[\]]{}::N & willow\
Welle::N & wave\
Welt::N & world\
Westen::N & west\
Wetter::N & weather\
Wiege::N & cradle\
Wiese::N & meadow\
Wind::N & wind\
Winkel::N & angle\
Woche::N & week\
Wolke::N & cloud\
Wolle::N & wool\
Wort::N & word\
Wunde::N & wound\
Wunsch::N & wish\
Wurzel::N & root\
Zahn::N & tooth\
Zaun::N & fence\
Zeh::N & toe\
Zeichen::N & sign\
Zeit::N & time\
Zeitung::N & newspaper\
Zunge::N & tongue\
Zweig::N & branch\
Zwiebel::N & onion\
[Ä]{}rmel::N & sleeve\
[Ö]{}l::N & oil\

[^1]: Whorf’s claim has subsequently been challenged. Later analyses of Hopi grammar suggest that the language marks two tenses: future and non-future [@malotki1983hopi].

[^2]: The hypothesis is named after both Benjamin Whorf and his Ph.D. advisor Edward Sapir.

[^3]: For notational clarity, we use different symbols for types and tokens. In this work, $v_i$ will always denote a word type, that is, the $i^\text{th}$ lexical item in a fixed vocabulary $V$, whereas $w_i$ will denote the $i^\text{th}$ token in a sentence $\mathbf{w}$.

[^4]: <http://cistern.cis.lmu.de/lemming/>

[^5]: <http://www.northeuralex.org/>
