Answers.Body: <p>Geoffrey Hinton has started working on Thought Vectors at Google: <a href="https://en.wikipedia.org/wiki/Thought_vector" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Thought_vector</a></p>
  
  <p>The basic idea is similar to his original idea with Capsule Networks, where activation happens by vectors instead of scalars, which allows the network to capture transformations: for example while traditional CNN needs to see object from all perspectives of three dimensional space, the Capsule networks are able to extrapolate transformations such as stretching much better.</p>
  
  <p>Thought Vectors guide NLP similarly; one could say that there are two grammars, the linguistic grammar and the narrative grammar which is more universal (Vladimir Propp, Joseph Campbell, John Vervake). While dependency grammars do great job at understanding linguistic grammar, we lack tools for meaning extraction, which is narrative bound. Thus Thought Vectors could, at least in theory, give us a framework for matching the meaning of a word within a context rather than just lexically and grammarly trying to approximate the meaning through average co-occurances.</p>
  
  <p>Neural Networks with Thought Vectors would be highly complex and beyond our computational resources today (Hinton predicts in one paper, that we would get there around 2035), however, one could conduct empirical research already by giving a heuristic structure for Thought Vectors by utilizing narrative systems that do compute more easily. One could for example have text segments annotated with writing theories or other such devices that would approximate the Thought Vectors conceptually. For example annotating the text with state transformations of conflict driven partially ordered causal link planner (cPOCL, Gervas et al.) or use a writing theory framework such as Dramatica to annotate known movie scripts (<a href="http://dramatica.com/theory" rel="nofollow noreferrer">http://dramatica.com/theory</a> <a href="http://dramatica.com/analysis" rel="nofollow noreferrer">http://dramatica.com/analysis</a>).</p>
  
  <p>Hinton himself is currently active in NLP research: <a href="https://research.google/people/GeoffreyHinton/" rel="nofollow noreferrer">https://research.google/people/GeoffreyHinton/</a></p>
  
  <p>Here is a nice explanation of Thought Vectors: <a href="https://pathmind.com/wiki/thought-vectors" rel="nofollow noreferrer">https://pathmind.com/wiki/thought-vectors</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-23T09:50:52.870Z
Answers.LastActivityDate: 2020-06-15T14:33:32.257Z
Answers.LastEditDate: 2020-06-15T14:33:32.257Z
Answers.UserDisplayName: Ahti Ahde
Answers.Body: <p>For us to answer this question. First, we need to look at why capsule networks outperform convolution neural networks by as much as 45% in recognizing images that have been rotated, translated or are under a different pose. We can find Geof Hinton's paper on capsule networks here for reference <a href="https://arxiv.org/pdf/1710.09829v1.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1710.09829v1.pdf</a></p>
  
  <p>In a CNN architecture, a convolution layer is usually followed by a max-pooling layer. This is so that the lower levels can detect low-level features, like edges, while the high-level layers can detect abstraction like eyes. However, the application of max-pooling leads to the loss of important information regarding the location and spatial relationship between certain features. </p>
  
  <p>On the other hand, this is where capsule networks excel, the way they represent certain features is locally invariant. This is why capsule networks can recognize images under different lighting conditions and deformations. They are likely to excel at applications such as video and object tracking but not necessarily NLP.</p>
  
  <p>The current approach in NLP maps words and phrases to vectors. From there, we exploit the concept of vectors and distances between them (cosine, euclidean, etc.) to perform operations such as: finding the similarity between words and even documents, machine translation, and natural language understanding (NLU). </p>
  
  <p>Capsule networks are unlikely to succeed in NLP. This is because algorithms that aim to find the hierarchical structure of natural languages or approaches that focus on grammar have met little success. Research by Stanford University aiming at finding the hierarchical structure of natural languages can be found here <a href="https://nlp.stanford.edu/projects/project-induction.shtml" rel="nofollow noreferrer">https://nlp.stanford.edu/projects/project-induction.shtml</a></p>
  
  <p>Although conclusive research regarding other applications of capsule networks has not yet been conducted. They are likely to excel at applications such as video intelligence and object tracking but not necessarily NLP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-15T14:39:01.677Z
Answers.Text: It seems that it no longer is true that "Capsule Networks are unlikely to succeed in NLP" as Hinton himself is conducting research on this at Google; thus I think I should uncorrect this answer. I do understand that conceptually it is hard to distinct where NLP ends and something else starts, however, I would believe Thought Vectors for example could provide a similar addition to NLP as Dependency Grammars; they could detect the difference of thematic meaningness of words far better than n-gram based methods.
Answers.UserDisplayName: Ahti Ahde
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-01-19T18:15:08.177Z
Answers.LastActivityDate: 2020-06-09T11:49:52.367Z
Answers.LastEditDate: 2020-06-09T11:49:52.367Z
Answers.UserDisplayName: Seth Simba
Answers.Body: <p>There has been some recent work on this: <a href="https://arxiv.org/pdf/1804.00538.pdf" rel="nofollow noreferrer">Investigating Capsule Networks with Dynamic Routing for Text Classification</a></p>
  
  <p>Seems some are having some success with it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-25T08:20:08.247Z
Answers.Text: Instead of just a link, it would be more useful if you could list the title of the paper and what it is about, so that people can assess the relevance before downloading it.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-26T16:19:54.643Z
Answers.Text: Well, it would be about Capsule networks and NLP.  That doesn't seem to be to great a conceptual leap for most people to make.
Answers.UserDisplayName: user2202800
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-22T19:47:32.280Z
Answers.LastActivityDate: 2020-06-09T11:51:01.577Z
Answers.LastEditDate: 2020-06-09T11:51:01.577Z
Answers.UserDisplayName: user2202800
Body: <p>The capsule neural network seems to be a good solution for problems that involve hierarchies. For example, a face is composed of eyes, a nose and ears; a hand is made of fingers, nails, and a palm; and a human is composed of a face and hands.</p>
  
  <p>Many problems in NLP can be seen as hierarchical problems: there are words, sentences, paragraphs, and chapters, whose meaning changes based on the style of lower levels.</p>
  
  <p>Are there any research papers (which I should be aware of) on the application of capsule neural networks to NLP problems? </p>
  
  <p>Are there related research papers, which have been investigating hierarchical complexity within the domain of NLP, which could be easily translated to Capsule Network?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-18T11:55:04.647Z
Comments.Text: I think there might be something like NNP "Natural Narrative Processing", which would be the superset of NLP.
Comments.UserDisplayName: Ahti Ahde
Tags: <neural-networks><natural-language-processing><applications><capsule-neural-network>
Title: Have capsule neural networks been used to NLP problems?
UserDisplayName: Ahti Ahde
ThreadText: # Have capsule neural networks been used to NLP problems?
  
  (Asked by: username_0 on 2017-12-16)
  
  The capsule neural network seems to be a good solution for problems that involve hierarchies. For example, a face is composed of eyes, a nose and ears; a hand is made of fingers, nails, and a palm; and a human is composed of a face and hands.
  
  Many problems in NLP can be seen as hierarchical problems: there are words, sentences, paragraphs, and chapters, whose meaning changes based on the style of lower levels.
  
  Are there any research papers (which I should be aware of) on the application of capsule neural networks to NLP problems?
  
  Are there related research papers, which have been investigating hierarchical complexity within the domain of NLP, which could be easily translated to Capsule Network?
  
  **Tags:** neural-networks, natural-language-processing, applications, capsule-neural-network
  
  > **Comments:**
  > **username_0**: I think there might be something like NNP "Natural Narrative Processing", which would be the superset of NLP.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2020-04-23. Score: 2 | Accepted answer
  
  Geoffrey Hinton has started working on Thought Vectors at Google: <https://en.wikipedia.org/wiki/Thought_vector>
  
  The basic idea is similar to his original idea with Capsule Networks, where activation happens by vectors instead of scalars, which allows the network to capture transformations: for example while traditional CNN needs to see object from all perspectives of three dimensional space, the Capsule networks are able to extrapolate transformations such as stretching much better.
  
  Thought Vectors guide NLP similarly; one could say that there are two grammars, the linguistic grammar and the narrative grammar which is more universal (Vladimir Propp, Joseph Campbell, John Vervake). While dependency grammars do great job at understanding linguistic grammar, we lack tools for meaning extraction, which is narrative bound. Thus Thought Vectors could, at least in theory, give us a framework for matching the meaning of a word within a context rather than just lexically and grammarly trying to approximate the meaning through average co-occurances.
  
  Neural Networks with Thought Vectors would be highly complex and beyond our computational resources today (Hinton predicts in one paper, that we would get there around 2035), however, one could conduct empirical research already by giving a heuristic structure for Thought Vectors by utilizing narrative systems that do compute more easily. One could for example have text segments annotated with writing theories or other such devices that would approximate the Thought Vectors conceptually. For example annotating the text with state transformations of conflict driven partially ordered causal link planner (cPOCL, Gervas et al.) or use a writing theory framework such as Dramatica to annotate known movie scripts (<http://dramatica.com/theory> <http://dramatica.com/analysis>).
  
  Hinton himself is currently active in NLP research: <https://research.google/people/GeoffreyHinton/>
  
  Here is a nice explanation of Thought Vectors: <https://pathmind.com/wiki/thought-vectors>
  
  
  ---
  
  ### Answer by username_1 on 2018-01-19. Score: 3
  
  For us to answer this question. First, we need to look at why capsule networks outperform convolution neural networks by as much as 45% in recognizing images that have been rotated, translated or are under a different pose. We can find Geof Hinton's paper on capsule networks here for reference <https://arxiv.org/pdf/1710.09829v1.pdf>
  
  In a CNN architecture, a convolution layer is usually followed by a max-pooling layer. This is so that the lower levels can detect low-level features, like edges, while the high-level layers can detect abstraction like eyes. However, the application of max-pooling leads to the loss of important information regarding the location and spatial relationship between certain features.
  
  On the other hand, this is where capsule networks excel, the way they represent certain features is locally invariant. This is why capsule networks can recognize images under different lighting conditions and deformations. They are likely to excel at applications such as video and object tracking but not necessarily NLP.
  
  The current approach in NLP maps words and phrases to vectors. From there, we exploit the concept of vectors and distances between them (cosine, euclidean, etc.) to perform operations such as: finding the similarity between words and even documents, machine translation, and natural language understanding (NLU).
  
  Capsule networks are unlikely to succeed in NLP. This is because algorithms that aim to find the hierarchical structure of natural languages or approaches that focus on grammar have met little success. Research by Stanford University aiming at finding the hierarchical structure of natural languages can be found here <https://nlp.stanford.edu/projects/project-induction.shtml>
  
  Although conclusive research regarding other applications of capsule networks has not yet been conducted. They are likely to excel at applications such as video intelligence and object tracking but not necessarily NLP.
  
  > **Comments:**
  > **username_0**: It seems that it no longer is true that "Capsule Networks are unlikely to succeed in NLP" as Hinton himself is conducting research on this at Google; thus I think I should uncorrect this answer. I do understand that conceptually it is hard to distinct where NLP ends and something else starts, however, I would believe Thought Vectors for example could provide a similar addition to NLP as Dependency Grammars; they could detect the difference of thematic meaningness of words far better than n-gram based methods.  
  
  ---
  
  ### Answer by username_2 on 2018-06-22. Score: 1
  
  There has been some recent work on this: [Investigating Capsule Networks with Dynamic Routing for Text Classification](https://arxiv.org/pdf/1804.00538.pdf)
  
  Seems some are having some success with it.
  
  > **Comments:**
  > **username_3**: Instead of just a link, it would be more useful if you could list the title of the paper and what it is about, so that people can assess the relevance before downloading it.  
  > **username_2**: Well, it would be about Capsule networks and NLP.  That doesn't seem to be to great a conceptual leap for most people to make.  
  
  ---
  
  