Learning representations of words in a continuous space is perhaps the most fundamental task in NLP, however words interact in ways much richer than vector dot product similarity can provide. Many relationships between words can be expressed set-theoretically, for example adjective-noun compounds (eg. red cars  cars) and homographs (eg. tongue  body should be similar to mouth, while tongue  language should be similar to dialect) have natural set-theoretic interpretations. Box embeddings are a novel region-based representation which provide the capability to perform these set-theoretic operations. In this work, we provide a fuzzy-set interpretation of box embeddings, and learn box representations of words using a set-theoretic training objective. We demonstrate improved performance on various word similarity tasks, particularly on less common words, and perform a quantitative and qualitative analysis exploring the additional unique expressivity provided by WORD 2B OX . The concept of learning a distributed representation for a word has fundamentally changed the field of natural language processing. The introduction of efficient methods for training vector representations of words in Word2Vec (Mikolov et al., 2013), and later GloVe (Pennington et al.) as well as Fast-Text (Bojanowski et al., 2017) revolutionized the field, paving the way for the recent wave of deep architectures for language modeling, all of which implicitly rely on this fundamental notion that a word can be effectively represented by a vector. While now ubiquitous, the concept of representing a word as a single point in space is not particularly natural. All senses and contexts, levels of abstraction, variants and modifications which the word may represent are forced to be captured by  *Equal Contributions. the specification of a single location in Euclidean space. It is thus unsurprising that a number of alternatives have been proposed. Gaussian embeddings (Vilnis and McCallum, 2015) propose modeling words using densities in latent space as a way to explicitly capture uncertainty. Poincar embeddings (Tifrea et al., 2019) attempt to capture a latent hierarchical graph between words by embedding words as vectors in hyperbolic space. Trained over large corpora via similar unsupervised objectives as vector baselines, these models demonstrate an improvement on word 2263 similarity tasks, giving evidence to the notion that vectors are not capturing all relevant structure from their unsupervised training objective. A more recent line of work explores region-based embeddings, which use geometric objects such as disks (Suzuki et al., 2019), cones (Vendrov et al., 2016; Lai and Hockenmaier, 2017; Ganea et al., 2018), and boxes (Vilnis et al., 2018) to represent entities. These models are often motivated by the need to express asymmetry, benefit from particular inductive biases, or benefit from calibrated probabilistic semantics. In the context of word representation, their ability to represent words using geometric objects with well-defined intersection, union, and difference operations is of interest, as we may expect these operations to translate to the words being represented in a meaningful way. In this work, we introduce WORD 2B OX , a region-based embedding for words where each word is represented by an n -dimensional hyperrect-angle or box. Of the region-based embeddings, boxes were chosen as the operations of intersection, union, and difference are easily calculable. Specifically, we use a variant of box embeddings known as Gumbel boxes, introduced in (Dasgupta et al., 2020). Our objective (both for training and inference) is inherently set-theoretic, not probabilistic, and as such we first provide a fuzzy-set interpretation of Gumbel boxes yielding rigorously defined mathematical operations for intersection, union, and difference of Gumbel boxes. We train boxes on a large corpus in an unsupervised manner with a continuous bag of words (CBOW) training objective, using the intersection of boxes representing the context words as the representation for the context. The resulting model demonstrates improved performance compared to vector baselines on a large number of word similarity benchmarks. We also compare the models' abilities to handle set-theoretic queries, and find that the box model outperforms the vector model 90% of the time. Inspecting the model outputs qualitatively also demonstrates that WORD 2B OX can provide sensible answers to a wide range of set-theoretic queries. Notation Let V = { v i } Ni =1 denote the vocabulary, indexed in a fixed but arbitrary order. A sentence s = ( s 1 , . . . , s j ) is simply a (variable-length) sequence of elements in our vocab s i  V . We view our corpus C = { s i } as a multiset 1 of all sentences in our corpus. Given some fixed window size  , for each word s i in a sentence s we can consider the window centered at i , w i = [ s i   , . . . , s i , . . . , s i +  ] , where we omit any indices exceeding the bounds of the sentence. Given a window w i we denote the center word using cen( w i ) = s i , and denote all remaining words as the context con( w i ) . We let CW be the multiset of all windows in the corpus. Given any ambient space U a set S  U can be represented by its characteristic function 1 S : U  { 0 , 1 } such that 1 S ( u ) = 1  u  S . This definition can be generalized to consider functions m : U  [0 , 1] , in which case we call the pair A = ( U, m ) a fuzzy set and m = m A is known as the membership function (Zadeh, 1965; Klir and Yuan, 1996). There is historical precedent for the use of fuzzy sets in computational linguistics (Zhelezniak et al., 2019a; Lee and Zadeh, 1969). More generally, fuzzy sets are naturally required any time we would like to learn a set representation in a gradient-based model, as hard membership assignments would not allow for gradient flow. In order to extend the notion of set intersection to fuzzy sets, it is necessary to define a t-norm , which is a binary operation  : [0 , 1]  [0 , 1]  [0 , 1] which is commutative, monotonic, associative, and equal to the identity when either input is 1. The min and product operations are common examples of t-norms. Given any t-norm, the intersection of fuzzy sets A and B has membership function m A  B ( x ) =  ( m A ( x ) , m B ( x )) . Any t-norm has a corresponding t-conorm which is given by  ( a, b ) = 1   (1  a, 1  b ) ; for min the t-conorm is max , and for product the t-conorm is the probabilistic sum,  sum ( a, b ) = a + b  ab . This defines the union between fuzzy sets, where m A  B ( x ) =  ( m A ( x ) , m B ( x )) . Finally, the complement of a fuzzy set simply has member function m A c ( x ) = 1  m A ( x ) . Box embeddings, introduced in (Vilnis et al., 2018), represent elements x of some set X as a Cartesian 1 A multiset is a set which allows for repetition, or equivalently a sequence where order is ignored. product of intervals, Box( x ) := d (cid:89) i =1 [ x  i , x + i ] = [ x  1 , x +1 ]      [ x  d , x + d ]  R d . (1) The volume of a box is simply the multiplication of the side-lengths, | Box( x ) | = d (cid:89) i =1 max(0 , x + i  x  i ) , and when two boxes intersect, their intersection is Box( x )  Box( y ) = d (cid:89) i =1 [max( x  i , y  i ) , min( x + i , y + i )] . Boxes are trained via gradient descent, and these hard min and max operations result in large areas of the parameter space with no gradient signal. Dasgupta et al. (2020) address this problem by modeling the corners of the boxes { x  i } with Gumbel random variables, { X  i } , where the probability of any point z  R d being inside the box Box G ( x ) is given by P ( z  Box G ( x )) = d (cid:89) i =1 P ( z i > X  i ) P ( z i < X + i ) . For clarity, we will denote the original (hard) boxes as Box , and the Gumbel boxes as Box G . The Gumbel distribution was chosen as it was min/max stable, thus the intersection Box G ( x )  Box G ( y ) which was defined as a new box with corners modeled by the random variables { Z  i } where Z  i := max( X  i , Y  i ) and Z + i := min( X + i , Y + i ) is actually a Gumbel box as well. Boratko et al. (2021) observed that P ( z  Box G ( x )  Box G ( y )) = P ( z  Box G ( x )) P ( z  Box G ( y )) , (2) and also provided a rigorous probabilistic interpretation for Gumbel boxes when embedded in a space of finite measure, leading to natural notions of union and intersection based on these operations of the random variables (Boratko et al., 2021). In this work, we do not embed the boxes in a space of finite measure, but instead interpret them as fuzzy sets , where the above probability (of a point z being inside the Gumbel box) acts as a soft membership function. In this section, we describe the motivation for using fuzzy sets to represent words, starting with an approach using traditional sets. First, given a word v  V , we can consider the windows centered at v , cen W ( v ) := { w  CW : cen( w ) = v } , and the set of windows whose context contains v , con W ( v ) := { w  CW : con( w )  v } . Note that cen W is a function which takes in a word and returns a set of windows, whereas cen is a function which takes in a window and returns the center word, and a similar distinction holds for con W and con . A given window is thus contained inside the intersection of the sets described above, namely [ w  j , . . . , w 0 , . . . , w j ]  cen W ( w 0 )  (cid:92) i  =0 con W ( w i ) . is contained inside the cen W ( fox ) set, as well as con W ( quick ) , con W ( brown ) , con W ( jumps ) , con W ( over ) . With this formulation, the intersection of the con W sets provide a natural choice of representation for the context. We might hope that cen W ( v ) provides a reasonable representation for the word v itself, however by our set theoretic definition for any u  = v we have cen W ( u )  cen W ( v ) =  . We would like the representation of u to overlap with v if u has similar meaning to v , i.e. we would like to consider (cid:93) cen W ( v ) := { w  W : cen( w ) similar to v } . A crisp definition of meaning or similarity is not possible (Hill et al., 2015; Finkelstein et al., 2001) due to individual subjectivity. Inter-annotator agreement for Hill et al. (2015) is only 0.67, for example, which makes it clear that (cid:93) cen W ( v ) could not possibly be represented as a traditional set. Instead, it seems natural to consider (cid:93) cen W ( v ) as represented by a fuzzy set ( W, m ) , where m ( w )  2265 [0 , 1] can be thought of as capturing graded similarity between v and cen( w ) . 2 In the same way, we can define (cid:94) con W ( v ) := { w  W : con( v )  w similar to v } , which would also be represented as a fuzzy set. 3 As we wish to capture these similarities with a machine learning model, we now must find trainable representations of fuzzy sets. Remark 1. Our objective of learning trainable representations for these sets provides an additional practical motivation for using fuzzy sets namely, the hard assignment of elements to a set is not differentiable. Any gradient-descent based learning algorithm which seeks to represent sets will have to consider a smoothed variant of the characteristic function, which thus leads to fuzzy sets. In this section we will describe how we model fuzzy sets using Gumbel boxes (Dasgupta et al., 2020). As noted in Section 2.2, the Gumbel Box model represents entities x  X by Box G ( x ) with corners modeled by Gumbel random variables { X  i } . The probability of a point z  R d being inside this box is P ( z  Box G ( x )) = d (cid:89) i =1 P ( z i > X  i ) P ( z i < X + i ) . Since this is contained in [0 , 1] , we have that ( R d , P ( z  Box G ( x )) is a fuzzy set. For clarity, we will refer to this fuzzy set as Box F ( x ) . The set complement operation has a very natural interpretation in this setting, as Box F ( x ) c has membership function 1  P ( z  Box G ( x )) , that is, the probability of z not being inside the Gumbel box. The product t-norm is a very natural choice as well, as the intersection Box F ( x )  Box F ( y ) will have membership function P ( z  Box G ( x )) P ( z  Box G ( y )) , which is precisely the membership function associated with Box G ( x )  Box G ( y ) , where here the intersection is between Gumbel boxes as defined in Dasgupta et al. (2020). Finally, we find that the membership function for 2 For an even more tangible definition, we can consider m ( w ) the percentage of people who consider u to be similar to cen( w ) when used in context con( w ) . 3 Note that this gives a principled reason to use different representation for (cid:94) cen W ( v ) and (cid:94) con W ( v ) , as they fundamentally represent different sets. Remark 2. Prior work on Gumbel boxes had not defined a union operation on Gumbel boxes, however (3) has several pleasing properties apart from being a natural consequence of using the product t-norm. First, it can be directly interpreted as the probability of z being inside Box G ( x ) or Box G ( y ) . Second, if the Gumbel boxes were embedded in a space of finite measure, as in Boratko et al. (2021), integrating (3) would yield the probability corresponding to P (Box( x )  Box( y )) . (cid:90) The connection between this integral and that which was approximated in Dasgupta et al. (2020) is provided by Lemma 3 of Boratko et al. (2021), and thus we have | Box F ( x ) |  d (cid:89) i =1  log (cid:18) 1 + exp (cid:18)  + i    i   2  (cid:19)(cid:19) where   i ,  + i are the location parameters for the Gumbel random variables X  i , X + i , respectively. As mentioned in Section 2.2, Gumbel boxes are closed under intersection, i.e. Box G ( x )  Box G ( y ) is also a Gumbel box, which implies that the size of the fuzzy intersection | Box F ( x )  Box F ( y ) | = (cid:90) R d P ( z  Box G ( x )) P ( z  Box G ( y )) d z = (cid:90) R d P ( z  Box G ( x )  Box G ( y )) d z can be approximated as well. As both of these are tractable, integrating (3) is also possible via linearity. Similarly, we can calculate the size of fuzzy set differences, such as | Box F ( x ) \ Box F ( y ) | = (cid:90) R d P ( z  Box G ( x ))[1  P ( z  Box G ( y ))] d z . By exploiting linearity and closure under intersection, it is possible to calculate the size of arbitrary fuzzy intersections, unions, and set differences, as well as any combination of such operations. Remark 3. If our boxes were embedded in a space of finite measure, as in Boratko et al. (2021), the sizes of these fuzzy sets would correspond to the intersection, union, and negation of the binary random variables they represent. In this section we describe our method of training fuzzy box representations of words, which we refer to as WORD 2B OX . In Section 3 we defined the fuzzy sets (cid:93) cen W ( v ) and (cid:93) cen W ( v ) , and in Section 4 we established that Gumbel boxes can be interpreted as fuzzy sets, thus for WORD 2B OX we propose to learn center and context box representations cen B ( v ) := Box F ( (cid:93) cen W ( v )) con B ( v ) := Box F ( (cid:94) con W ( v )) . Given a window, w = [ w  j , . . . , w 0 , . . . , w j ] , we noted that w must exist in the intersection, (cid:93) cen W ( w 0 )  (cid:92) i  =0 (cid:94) con W ( w i ) (4) and thus we consider a max-margin training objective where the score for a given window is given as f ( w ) := (cid:12)(cid:12)(cid:12)(cid:12) cen B ( w 0 )  (cid:92) i  =0 con B ( w i ) (cid:12)(cid:12)(cid:12)(cid:12) . (5) To create a negative example w  we follow the same procedure as CBOW from Mikolov et al. (2013), replacing center words with a word sampled from the unigram distribution raised to the 3 / 4 . We also subsample the context words as in Mikolov et al. (2013). As a vector baseline, we compare with a WORD 2V EC model trained in CBOW-style. We attach the source code with supplementary material. We evaluate both WORD 2V EC and WORD 2B OX on several quantitative and qualitative tasks that cover the aspects of semantic similarity, relatedness, lexical ambiguity, and uncertainty. Following the previous relevant works (Athiwaratkun and Wilson, 2018; Meyer and Lewis, 2020; Baroni et al., 2012), we train on the lemmatized WaCkypedia corpora (Baroni et al., 2009), specifically ukWaC which is an English language corpus created by web crawling. After additional pre-processing (details in appendix A) the corpus contains around 0.9 billion tokens, with just more than 112k unique tokens in the vocabulary. Noting that an n -dimensional box actually has 2 n parameters (for min and max coordinates), we compare 128-dimensional WORD 2V EC embeddings and 64-dimensional WORD 2B OX embeddings for all our experiments. We train over 60 different models for both the methods for 10 epochs using random sampling on a wide range of hyperparameters (please refer to appendix C for details including learning rate, batch size, negative sampling, sub-sampling threshold, etc.). In order to ensure that the only difference between the models was the representation itself, we implemented a version of WORD 2V EC in PyTorch, including the negative sampling and subsampling procedures recommended in (Mikolov et al., 2013), using the original implementation as a reference. As we intended to train on GPU, however, our implementation differs from the original in that we use Stochastic Gradient Descent with varying batch sizes. We provide our source code at https://github.com/iesl/word2box . We primarily evaluate our method on several word similarity benchmarks: SimLex-999 (Hill et al., 2015), WS-353 (Finkelstein et al., 2001), YP-130 (Yang and Powers, 2006), MEN (Bruni et al., 2014), MC-30 (Miller and Charles, 1991), RG-65 (Ruben-stein and Goodenough, 1965), VERB-143 (Baker et al., 2014), Stanford RW (Luong et al., 2013), Mturk-287 (Radinsky et al., 2011) and Mturk-771 (Halawi et al., 2012). These datasets consist of pairs of words (both noun and verb pairs) that are annotated by human evaluators for semantic similarity and relatedness. In table 1 we compare the WORD 2B OX and WORD 2V EC models which perform best on the similarity benchmarks. We observe that WORD 2B OX outperforms WORD 2V EC (as well as the results reported by other baselines) in the majority of the word similarity tasks. We outperform WORD 2V EC by a large margin in Stanford RW and YP-130, which are the rare-word datasets for noun and verb respectively. Noticing this effect, we enumerated the frequency distribution of each dataset. The datasets fall in different sections of the frequency spectrum, e.g., Stanford RW (Lu-ong et al., 2013) only contains rare words which make its median frequency to be 5,683, whereas 2267 Figure 2: This plot depicts the gain in correlation score for WORD 2B OX against WORD 2V EC is much higher for the low and mid frequency range. WS-353 (Rel) (Finkelstein et al., 2001) contains many more common words, with a median frequency of 64,490. We also observe a larger relative performance improvement over WORD 2V EC on other datasets which have low to median frequency words, e.g. MC-30, MEN-Tr-3K, and RG-65, all with median frequency less than 25k. The order they appear in the table and the subsequent plots is lowest to highest frequency, left to right. Please refer to Appendix B for details. In figure 2, we see that WORD 2B OX outperforms WORD 2V EC more significantly with less common words. In order to investigate further, we selected four datasets (RW-Stanford (rare words), Simelex-999, SimVerb-3500,WS-353 (Rel)), truncated them at a frequency threshold, and calculated the correlation for different levels of this threshold. In figure 3, we demonstrate how the performance gap between WORD 2B OX and WORD 2V EC changes as increasing amount frequent words are added to these similarity datasets. We posit that the geometry of box embeddings is more flexible in the way it handles sets of mutually disjoint words (such as rare words) which all co-occur with a more common word. Boxes have exponentially many corners, relative to their dimension, allowing extreme flexibility in the possible arrangements of intersection to represent complicated co-occurrances. All the senses, contexts and abstractions of a word can not be captured accurately using a point vector, and must be captured with sets. In this section, we evaluate our models capability of representing sets by performing set operations with the trained models. Homographs, words with identical spelling but distinct meanings, and polysemous words are ideal choice of probe for this purpose, as demonstrated by the bank, river and finance example of Figure 1. We constructed set-theoretic logical operations on words based on common polysemous words and homographs (Nelson et al., 1980). For example, the word property will have association with words related both asset and attribute, and thus the union of the later two should be close to the original word property. Likewise, the intersection set of property and math should contain many words related to mathematical properties or concepts. To this end, we created a dataset consisting of triples ( A, B, C ) where A  B should yield a set similar to C , for various set-theoretic operations  . In this task, given two words A and B and a set theoretic operation  , we try to find the rank of word C in the sorted list based on the set similarity (vector similarity scores for the vectors) score between A  B and all words in the vocab. The dataset consists of 52 examples for both Union and Negation, and 20 examples for Intersection. The details of the dataset can be found in Appendix D. 6.2.2 Quantitative Results In Table 2, we report the percentage of times WORD 2B OX outperforms WORD 2V EC , i.e. the model yields better rank for the word C . Note that it is not clear how to design the union, difference or the intersection operations with vectors. We consider several relevant choices, including component-wise operations (addition, subtraction, min and max) which yield a representation for A  B , as well as operations which operate on the scores eg. score max pooling ranks each word X using max( A  X, B  X ) , and similarly for score min pooling. The purpose of these operations is to mimic the essence of union and intersection in the vector space, however, it is evident that the trained vector geometry is not harmonious to this construction as well. We observe that almost of all the values are more than 0.9, meaning that WORD 2B OX yields a higher rank for the target C than WORD 2V EC over 90% of the time. This empirically validates that our model is indeed capturing the underlying set theoretic aspects of the words in the corpus. In this section, we present some interesting examples of set theoretic queries on words, with different degrees of complexities. For all the tables in this section, we perform the set-operations on the query words and present the ranked list of most similar words to the output query. Many of these queries are based on the aforementioned homographs, for which there are natural expectations of what various set-theoretic operations should capture. Our results are presented in Table 3-7. The results in Table 4 look reasonable for both models, as is to be expected since this is simply the similarity function for each model. Even increasing to a single intersection, as in Table 5, starts to demonstrate that WORD 2V EC may often return very low-frequency words. In Table 6, we observe that set difference of property and land yields a set of words that are related to attributes of science subjects, eg. algebra or chemistry. We wanted to examine how the model would handle more complicated queries, for example if we first perform property \ finance and then further intersect with algebra or chemistry, does the introduction of the relatively high-frequency finance term cause the model to struggle to recapture these items? In Table 7 we observe that the outputs for WORD 2B OX do indeed correspond to properties of those sub-fields of science, whereas the results in WORD 2V EC focus strongly on finance. In general, we observe better consistency of WORD 2B OX with all the example logical queries. Learning distributional vector representations from a raw corpus was introduced in Mikolov et al. (2013), quickly followed by various improvements (Pennington et al.; Bojanowski et al., 2017). More recently, vector representations which incorporate contextual information have shown significant improvements (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020). As these models require context, however, Word2Vec-style approaches are still relevant in settings where such context is unavailable. Hyperbolic representations (Nickel and Kiela, 2017; Ganea et al., 2018; Chamberlain et al., 2017) have become popular in recent years. Most related to our setting, Tifrea et al. (2019) propose a hyperbolic analog to GloVe, with the motivation that the hyperbolic embeddings will discover a la-2269 WORD 2B OXWORD 2V EC ( bank  river )  X ( bank  river )  X ( bank \ river )  X ( bank + river )  X ( bank  river )  X max( bank , river )  X max( bank , river )  X max( bank  X, river  X ) min( bank  X, river  X ) headwaters tributary barclays tributaries cheques tributary vipava tributaries gauley tributary valley hsbc tributary tymoshenko tributaries quabbin headwaters pymatuning lake headwaters banking headwaters receivables prut irwell tributary 'utricularia basin reservoir citigroup nakdong citibank chambal trabajadores headwater luangwa estuary gorge citibank vipava eurozone headwaters chattahoochee distributaries vipava creek lake firm estuary brinks larrys tributaries larrys guadalquivir valley dam ipo larrys defrauded nakdong belait kobuk suir reservoir headwater brokerage headwater courtaulds waterway bougouriba estuary meenachil canal junction interbank distributary refinance loyalsock canal ijssel tributary floodplain creek kpmg luangwa mortgage 'hyperolius glomma distributary battuta Table 3: Output of WORD 2B OX and WORD 2V EC for various set operations WORD 2B OXWORD 2V ECAA  X A  X bank capital settlement airline hotel gateway treasury firm government loan casino debit depositors securities kaupthing interbank subprime counterparty citibank fdic nasdaq economics education architecture politics economy literature faculty agriculture phd journalism microeconomic keynesian microeconomics minored macroeconomics econometrics sociology thermodynamics evolutionism structuralist microeconomics economics mathematics physics philosophy theory technology economist principle research analysis microeconomic initio germline instantiation zachman macroeconomics oxoglutarate glycemic noncommutative pubmed property land register status manor purpose locality premise landmark site residence easement infringes burgage krajobrazowy chattels policyholder leasehold intestate liabilities ceteris rock music pop mountain cave band blues dance groove hot disco shoegaze rhyolitic punk britpop mafic outcrops metalcore bluesy sedimentary quartzite Table 4: Similarity outputs for WORD 2B OX and WORD 2V ECWORD 2B OXWORD 2V ECA B ( A  B )  X ( A + B )  X girl boy kid girls schoolgirl teenager woman boys child baby teenage orphan shoeshine nanoha soulja schoolgirl yeller beastie jeezy crudup 'girl rahne property burial cemetery bury estate grave interment tomb dwelling site gravesite sarcophagus interment moated interred dunams ceteris burials catafalque easement deeded inhumation historical historic estate artifact archaeological preserve ownership patrimony heritage landmark site krajobrazowy burgage easement kravis dilapidation tohono intangible domesday moated laertius house estate mansion manor residence houses tenement building premise buildings site leasehold mansion tenements outbuildings estate burgage bedrooms moated burgesses manor tongue body eye mouth ear limb lip forehead anus neck finger penis tubercle ribcage meatus diverticulum forelegs radula tuberosity elastin foramen nostrils language dialect idiom pronunciation meaning cognate word accent colloquial speaking speak fluently dialects vowels patois languages loanwords phonology lingala tigrinya fluent Table 5: Comparison of set intersection operation WORD 2B OXWORD 2V ECA B ( A \ B )  X ( A  B )  X algebra finance homomorphism isomorphism automorphism abelian algebraic bilinear topological morphism spinor homeomorphism homeomorphic unital homomorphisms nilpotent algebraically projective holomorphic propositional nondegenerate endomorphism bank finance wensum junction neman mouth tributary downstream corner embankment forks sandwich shaddai takla thrombus gauley paria epenthetic chibchan urubamba foremast bolshaya river barclays hsbc banking citigroup citibank firm ipo brokerage interbank kpmg cheques tymoshenko receivables citibank eurozone brinks defrauded courtaulds refinance mortgage chemistry finance biochemistry superconductor physics physic eutectic heat isotope fluorescence yttrium spectroscopy augite alkyne desorption phosphorylating dimorphism fumarate hypertrophic empedocles hydratase enantiomer property land homotopy isomorphism involution register bijection symplectic eigenvalue idempotent compactification lattice brst stieltjes l'p repressor absurdum doesn conjugates nonempty didn wouldn Table 6: Comparison of set difference operation 2270 WORD 2B OXWORD 2V ECA B C (( A \ B )  C )  X ( A  B + C )  X property finance algebra laplacian nilpotent antiderivative lattice surjective automorphism invertible homotopy integer integrand expropriate extort refco underwrite reimburse refinance parmalat refinancing brokerage privatizing chemistry eutectic desiccant allotrope phenocryst hardness solubility monoclinic hygroscopic nepheline trehalose refinance brokerage burgage stockbroking refinancing warranties reimburse madoff privatizing valorem Table 7: Comparison of set difference followed by intersection operation tent hierarchical structure between words. 4 Vilnis and McCallum (2015) use Gaussian distributions to represent each word, and KL Divergence as a score function. 5 Athiwaratkun and Wilson (2018) extended such representations by adding certain thresholds for each distribution. For a different purpose, Ren and Leskovec (2020) use Beta Distributions to model logical operations between words. Our work can be seen as a region-based analog to these models. Of the region-based embeddings, Suzuki et al. (2019) uses hyperbolic disks, and Ganea et al. (2018) uses hyperbolic cones, however these are not closed under intersection nor are their intersections easily computable. Vendrov et al. (2016) and Lai and Hockenmaier (2017) use an axis-aligned cone to represent a specific relation between words/sentences, for example an entailment relation. Vilnis et al. (2018) extends Lai and Hockenmaier (2017) by adding an upper-bound, provably increasing the representational capacity of the model. Li et al. (2019) and Dasgupta et al. (2020) are improved training methods to handle the difficulties inherent in gradient-descent based region learning. Ren et al. (2020) and Abboud et al. (2020) use a box-based adjustment of their loss functions, which suggest learning per-entity thresholds are beneficial. Chen et al. (2021) use box embeddings to model uncertain knowledge graphs, Onoe et al. (2021) use boxes for fined grained entity typing, and Patel et al. (2022) use boxes for multi-label classification. Fuzzy sets, a generalization of sets, have been widely studied in the context of clustering (Bezdek and Harris, 1978), decision theory (Zimmermann, 1987) and linguistics (De Cock et al., 2000). However, the use of fuzzy sets in NLP has been fairly limited. Bhat et al. (2020) normalized each dimension of a word vector against all the word vectors 4 Reported results are included in table 1 as Poincar 5 Reported results are included in table 1 as Gaussian in the vocabulary and interpret them as probability features that enabled them to perform fuzzy set theoretic operations with the words. Zhao and Mao (2018) and Zhelezniak et al. (2019b) build fuzzy set representations of sentences using pre-trained vector embeddings for words and show the usefulness such representations on semantic textual similarity (STS) tasks. Jimenez et al. (2013, 2014) use the soft-cardinality features for a fuzzy set representation of a sentence to perform the task of entailment and textual relatedness. All these works, use pre-trained vector embeddings for the words to form fuzzy sets representing sentences. However, in this work we learn fuzzy set representations for words from corpus. In this work we have demonstrated that box embeddings can not only effectively train to represent pairwise similarity but also can capture the rich set-theoretic structure of words via unsupervised training. This is a consequence of the fact that Gumbel boxes are an efficient parameterization of fuzzy sets, with sufficient representational capacity to model complicated co-occurrance interactions while, at the same time, allowing for tractable computation and gradient-based training of set-theoretic queries. The set-theoretic representation capabilities of box models allow them to generalize in a calibrated manner, leading to a more coherent and self-consistent model of sets. The authors would like to thank the members of the Information and Extraction Synthesis Laboratory (IESL) at UMass Amherst for helpful discussions. This work was partially supported by IBM Research AI through the AI Horizons Network and the Chan Zuckerberg Initiative under the project Scientific Knowledge Base Construction. Additional support was provided by the National 2271 Science Foundation (NSF) under Grant Numbers IIS-1514053 and IIS-2106391, the Defense Advanced Research Projects Agency (DARPA) via Contract No. FA8750-17-C-0106 under Subaward No. 89341790 from the University of Southern California, and the Office of Naval Research (ONR) via Contract No. N660011924032 under Subaward No. 123875727 from the University of Southern California. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IBM, CZI, NSF, DARPA, ONR, or the U.S. Government.
abstain abstain abstain objective objective abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method objective objective abstain result abstain other abstain abstain abstain method abstain other method other abstain other other other other other other other other other other other other abstain other other method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method method other other method abstain method abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain result result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain abstain result result other other other other other other other other abstain other other other other other other other other other other other other abstain objective abstain abstain other other other other other other other