Answers.Body: <p>I am not totally sure that this solution would work for your problem, however I'll take a stab at it, and hopefully this helps!</p>
  <p>One of my favorite resources so far for learning about transformers is this post <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="nofollow noreferrer">http://nlp.seas.harvard.edu/2018/04/03/attention.html</a>. I was able to follow along in a jupyter notebook and the only changes I remember I had to make to get it working were removing references to Variable. There is a section in there called <strong>Positional Encoding</strong> which may be relevant for what you want to do. Some people alternatively call this Positional Embedding. The choice of positional encoding is somewhat flexible. I have seen projects, such as Enformer from DeepMind (<a href="https://github.com/deepmind/deepmind-research/tree/master/enformer" rel="nofollow noreferrer">https://github.com/deepmind/deepmind-research/tree/master/enformer</a>), which use custom position embeddings for the DNA sequence domain, which is centered in the middle of a sequence and extends out to the sides.</p>
  <p>The above implementation/paper was very helpful for learning. Once I was ready to implement a transformer for a project I was working on, I found this implementation in the pytorch examples package a great place to start. In there you can see the class for the positional embedding and modify that as needed. <a href="https://github.com/pytorch/examples/tree/main/word_language_model" rel="nofollow noreferrer">https://github.com/pytorch/examples/tree/main/word_language_model</a></p>
  <p>Now for your specific problem I did some googling for positional embedding/encoding and graph transformer, and I found a project that looks promising <a href="https://github.com/inria-thoth/GraphiT" rel="nofollow noreferrer">https://github.com/inria-thoth/GraphiT</a>. Maybe that just does what you want, and their paper <a href="https://arxiv.org/abs/2106.05667" rel="nofollow noreferrer">https://arxiv.org/abs/2106.05667</a> may have some useful background for you.</p>
  <p>Hopefully these pointers help in your search!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T00:13:15.700Z
Answers.Text: could there be different kind of edges using "Positional Embedding?
Answers.UserDisplayName: Exploring
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-08T03:47:25.413Z
Answers.LastActivityDate: 2022-04-08T03:53:07.427Z
Answers.LastEditDate: 2022-04-08T03:53:07.427Z
Answers.UserDisplayName: John St. John
Body: <p>I am performing a Neural Machine Translation (NMT) task. In my case, input data has relational information.</p>
  <p>I know I can use a Graph Neural Network (GNN) and use a Graph2Seq model. But I can't find a good generational model for GNN.</p>
  <p>So I want to use Transformer. But then the challenge is how can I embed structural information there? Is there any open source artefact for Relational Transformer that I can use out of the box?</p>
  <p>Any pointers?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-04T01:30:44.437Z
Comments.Text: can you clarify the "relational information"?
Comments.UserDisplayName: CuCaRot
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-05T09:59:19.150Z
Comments.Text: Two nodes are connected by an edge with a relation. Think about a knowledge graph.
Comments.UserDisplayName: Exploring
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-06T03:58:44.410Z
Comments.Text: Would it make sense to define a positional embedding that is a function of the distance between nodes in your graph representation?
Comments.UserDisplayName: John St. John
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-06T04:09:47.043Z
Comments.Text: @JohnSt.John can you please explain in a bit more detail in an answer. Also can you please refer to an implementation if possible.
Comments.UserDisplayName: Exploring
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-08T03:47:38.497Z
Comments.Text: Done, hopefully that was helpful!
Comments.UserDisplayName: John St. John
Tags: <deep-learning><transformer><graph-neural-networks><gnn>
Title: How to use structural information in a Transformer?
UserDisplayName: Exploring
ThreadText: # How to use structural information in a Transformer?
  
  (Asked by: username_0 on 2022-03-29)
  
  I am performing a Neural Machine Translation (NMT) task. In my case, input data has relational information.
  
  I know I can use a Graph Neural Network (GNN) and use a Graph2Seq model. But I can't find a good generational model for GNN.
  
  So I want to use Transformer. But then the challenge is how can I embed structural information there? Is there any open source artefact for Relational Transformer that I can use out of the box?
  
  Any pointers?
  
  **Tags:** deep-learning, transformer, graph-neural-networks, gnn
  
  > **Comments:**
  > **username_1**: can you clarify the "relational information"?  
  > **username_0**: Two nodes are connected by an edge with a relation. Think about a knowledge graph.  
  > **username_2**: Would it make sense to define a positional embedding that is a function of the distance between nodes in your graph representation?  
  > **username_0**: @JohnSt.username_2 can you please explain in a bit more detail in an answer. Also can you please refer to an implementation if possible.  
  > **username_2**: Done, hopefully that was helpful!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-04-08. Score: 1
  
  I am not totally sure that this solution would work for your problem, however I'll take a stab at it, and hopefully this helps!
  
  One of my favorite resources so far for learning about transformers is this post <http://nlp.seas.harvard.edu/2018/04/03/attention.html>. I was able to follow along in a jupyter notebook and the only changes I remember I had to make to get it working were removing references to Variable. There is a section in there called **Positional Encoding** which may be relevant for what you want to do. Some people alternatively call this Positional Embedding. The choice of positional encoding is somewhat flexible. I have seen projects, such as Enformer from DeepMind (<https://github.com/deepmind/deepmind-research/tree/master/enformer>), which use custom position embeddings for the DNA sequence domain, which is centered in the middle of a sequence and extends out to the sides.
  
  The above implementation/paper was very helpful for learning. Once I was ready to implement a transformer for a project I was working on, I found this implementation in the pytorch examples package a great place to start. In there you can see the class for the positional embedding and modify that as needed. <https://github.com/pytorch/examples/tree/main/word_language_model>
  
  Now for your specific problem I did some googling for positional embedding/encoding and graph transformer, and I found a project that looks promising <https://github.com/inria-thoth/GraphiT>. Maybe that just does what you want, and their paper <https://arxiv.org/abs/2106.05667> may have some useful background for you.
  
  Hopefully these pointers help in your search!
  
  > **Comments:**
  > **username_0**: could there be different kind of edges using "Positional Embedding?  
  
  ---
  
  