What dimensions do the considered embeddings have?
Answer with content missing: (Models sections) 100, 200 and 400
We argue that extrapolation to examples outside the training space will often be easier for models that capture global structures, rather than just maximise their local fit to the training data. We show that this is true for two popular models: the Decomposable Attention Model and word2vec.
In a controversial essay, BIBREF0 draws the distinction between two types of generalisation: interpolation and extrapolation; with the former being predictions made between the training data points, and the latter being generalisation outside this space. He goes on to claim that deep learning is only effective at interpolation, but that human like learning and behaviour requires extrapolation. On Twitter, Thomas Diettrich rebutted this claim with the response that no methods extrapolate; that what appears to be extrapolation from X to Y is interpolation in a representation that makes X and Y look the same.  It is certainly true that extrapolation is hard, but there appear to be clear real-world examples. For example, in 1705, using Newton's then new inverse square law of gravity, Halley predicted the return of a comet 75 years in the future. This prediction was not only possible for a new celestial object for which only a limited amount of data was available, but was also effective on an orbital period twice as long as any of those known to Newton. Pre-Newtonian models required a set of parameters (deferents, epicycles, equants, etc.) for each body and so would struggle to generalise from known objects to new ones. Newton's theory of gravity, in contrast, not only described celestial orbits but also predicted the motion of bodies thrown or dropped on Earth. In fact, most scientists would regard this sort of extrapolation to new phenomena as a vital test of any theory's legitimacy. Thus, the question of what is required for extrapolation is reasonably important for the development of NLP and deep learning.  BIBREF0 proposes an experiment, consisting of learning the identity function for binary numbers, where the training set contains only the even integers but at test time the model is required to generalise to odd numbers. A standard multilayer perceptron (MLP) applied to this data fails to learn anything about the least significant bit in input and output, as it is constant throughout the training set, and therefore fails to generalise to the test set. Many readers of the article ridiculed the task and questioned its relevance. Here, we will argue that it is surprisingly easy to solve Marcus' even-odd task and that the problem it illustrates is actually endemic throughout machine learning.  BIBREF0 links his experiment to the systematic ways in which the meaning and use of a word in one context is related to its meaning and use in another BIBREF1 , BIBREF2 . These regularities allow us to extrapolate from sometimes even a single use of a word to understand all of its other uses. In fact, we can often use a symbol effectively with no prior data. For example, a language user that has never have encountered the symbol Socrates before may nonetheless be able to leverage their syntactic, semantic and inferential skills to conclude that Socrates is mortal contradicts Socrates is not mortal. Marcus' experiment essentially requires extrapolating what has been learned about one set of symbols to a new symbol in a systematic way. However, this transfer is not facilitated by the techniques usually associated with improving generalisation, such as L2-regularisation BIBREF3 , drop-out BIBREF4 or preferring flatter optima BIBREF5 . In the next section, we present four ways to solve this problem and discuss the role of global symmetry in effective extrapolation to the unseen digit. Following that we present practical examples of global structure in the representation of sentences and words. Global, in these examples, means a model form that introduces dependencies between distant regions of the input space.