I guess I should&#x27;ve connected the dots a bit better on this.<p>The neural network model offers a pretty compelling answer to this. The idea is that you each word is represented as a real-valued vector, and these vectors encode information about the word&#x27;s meaning. So, &quot;pizza&quot; encodes in it that it&#x27;s a type of food, and so does &quot;anchovies&quot;. But &quot;fork&quot; is closer to &quot;knife&quot; and &quot;spoon&quot;.<p>Linear models struggle with this. In a linear model, mostly, each word is an island. &quot;fork&quot; and &quot;spoon&quot; each have unique IDs, and so you&#x27;d better hope you see enough examples of each of them to understand what they mean. It helps to add semantic cluster IDs to the words as well, but this isn&#x27;t quite as good as the word vectors.<p>What doesn&#x27;t work particularly well is encoding external knowledge explicitly. Having a semantic resource that tells you explicitly that &quot;fork&quot; and &quot;spoon&quot; are members of class &quot;cutlery&quot;. This has been tried by many people, and it produces small or no benefit.