Answers.Body: <h2>An Image is Worth 16X16 Words:</h2>
  <h4>TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE</h4>
  <p>A Transformer consists of alternating layers of multiheaded self-attention.</p>
  <p>The <a href="https://arxiv.org/pdf/2010.11929.pdf" rel="nofollow noreferrer">Transformer Paper</a> adapts a NLP architecture for making Image Classification. For that, it first need to tokenize the image (like a piece of text). The tokenization is done by splitting the image into fixed-size patches and then embedding it.</p>
  <p>In other words, they treat a picture as a set of sub-images, just like a phrase is a set of words.</p>
  <h2>Relational Deep Reinforcement Learning</h2>
  <p>The <a href="https://arxiv.org/pdf/1806.01830.pdf" rel="nofollow noreferrer">Reinforcement Learning paper</a> uses self-attention to <strong>iteratively</strong> reason about the relations between entities in a scene and to guide a model-free policy.</p>
  <p>Focused on game playing, they decide to model a non-local concept of entities (players, objects) and the relationship between them.</p>
  <p>So instead of tokenizing the image, they first extract entities from it, using a more standard approach, which is convolution. So the multi-head attention layer receives a set of entities, the output is normalized so it's once again formatted as the entity set, so it can proceed with an iterative process.</p>
  <hr />
  <p>TL;DR:
  Same base (very promising) technique, but applied to 2 drastically different models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-24T08:28:44.507Z
Answers.LastActivityDate: 2021-08-24T17:15:54.447Z
Answers.LastEditDate: 2021-08-24T17:15:54.447Z
Answers.UserDisplayName: Andre Goulart
Body: <p>I am trying to figure out the difference between the architecture used in <a href="https://arxiv.org/pdf/2010.11929.pdf" rel="nofollow noreferrer">this</a> and <a href="https://arxiv.org/pdf/1806.01830.pdf" rel="nofollow noreferrer">this</a> paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-24T03:18:18.527Z
Comments.Text: I'm on top of the vision transformer but not the RL. But I know enough to tell you that these are very unrelated papers (within deep learning). And transformers are used across many types of tasks (vision, NLP, RL, audio).
  Imagine making an analogy from DL to all vehicles. In this context you can't exactly say a car is like a plane because they both have pistons. Same reason I wouldn't say these papers are particularly related by the fact that they both use transformers.
Comments.UserDisplayName: Alexander Soare
Tags: <comparison><papers><transformer><architecture><vision-transformer>
Title: What is the difference between a vision transformer and image-based relational learning?
UserDisplayName: desert_ranger
ThreadText: # What is the difference between a vision transformer and image-based relational learning?
  
  (Asked by: username_0 on 2021-08-18)
  
  I am trying to figure out the difference between the architecture used in [this](https://arxiv.org/pdf/2010.11929.pdf) and [this](https://arxiv.org/pdf/1806.01830.pdf) paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.
  
  **Tags:** comparison, papers, transformer, architecture, vision-transformer
  
  > **Comments:**
  > **username_1**: I'm on top of the vision transformer but not the RL. But I know enough to tell you that these are very unrelated papers (within deep learning). And transformers are used across many types of tasks (vision, NLP, RL, audio). Imagine making an analogy from DL to all vehicles. In this context you can't exactly say a car is like a plane because they both have pistons. Same reason I wouldn't say these papers are particularly related by the fact that they both use transformers.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-24. Score: 2 | Accepted answer
  
  ## An Image is Worth 16X16 Words:
  
  #### TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE
  
  A Transformer consists of alternating layers of multiheaded self-attention.
  
  The [Transformer Paper](https://arxiv.org/pdf/2010.11929.pdf) adapts a NLP architecture for making Image Classification. For that, it first need to tokenize the image (like a piece of text). The tokenization is done by splitting the image into fixed-size patches and then embedding it.
  
  In other words, they treat a picture as a set of sub-images, just like a phrase is a set of words.
  
  ## Relational Deep Reinforcement Learning
  
  The [Reinforcement Learning paper](https://arxiv.org/pdf/1806.01830.pdf) uses self-attention to **iteratively** reason about the relations between entities in a scene and to guide a model-free policy.
  
  Focused on game playing, they decide to model a non-local concept of entities (players, objects) and the relationship between them.
  
  So instead of tokenizing the image, they first extract entities from it, using a more standard approach, which is convolution. So the multi-head attention layer receives a set of entities, the output is normalized so it's once again formatted as the entity set, so it can proceed with an iterative process.
  
  ---
  
  TL;DR:
  Same base (very promising) technique, but applied to 2 drastically different models.
  
  
  ---
  
  