Answers.Body: <p>Yes, they can handle sequences with arbitrary length sequence, but with some remarks.</p>
  <p>In the paper <a href="https://arxiv.org/abs/2012.12877v2" rel="noreferrer">Training data-efficient image transformers &amp; distillation through attention</a> authors train models in the resolution 224x224 (1 + 14x14 tokens) and then finetune to the 384x384 (1 + 28x28 tokens).</p>
  <p>Weights to produce queries, keys, values, as well as feedforward layers, operate only on a single token and are agnostic to the sequence length.</p>
  <p>However, the size of the sequence is required in the positional embeddings, where one has a specific weight for each location of the token.</p>
  <p>In order to make this construction work for inputs of other sizes, one needs to transform this positional embedding in a certain way. <strong>Bicubic interpolation</strong> of positional embeddings, used in DeiT, works pretty well.
  One can use simpler <strong>Bilinear</strong> or <strong>Nearest</strong> interpolation - but it seems like this harms accuracy.</p>
  <p>In my own experience, when I took the DeiT-tiny with base accuracy of <strong>72.2%</strong> after several epochs of finetuning gave accuracy of <strong>77.0%</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-04T06:49:49.410Z
Answers.LastActivityDate: 2021-11-04T06:49:49.410Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>Does ViT do handle arbitrary sequence lengths using masking the same way the normal Transformer does?</p>
  <p>The <a href="https://arxiv.org/abs/2010.11929" rel="noreferrer">ViT paper</a> doesn't mention anything about it, so I assume it uses masking like the normal Transformer.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-03T23:56:37.407Z
Comments.Text: The code for that paper (among others) is located [here](https://github.com/google-research/vision_transformer). If someone is familiar with Tensorflow and Python then they should be able to figure out the answer to your question (note that you can also answer it yourself if you read through the code and discover it).
Comments.UserDisplayName: user
Tags: <neural-networks><computer-vision><transformer><vision-transformer>
Title: Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?
UserDisplayName: Dean R
ThreadText: # Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?
  
  (Asked by: username_0 on 2021-11-03)
  
  Does ViT do handle arbitrary sequence lengths using masking the same way the normal Transformer does?
  
  The [ViT paper](https://arxiv.org/abs/2010.11929) doesn't mention anything about it, so I assume it uses masking like the normal Transformer.
  
  **Tags:** neural-networks, computer-vision, transformer, vision-transformer
  
  > **Comments:**
  > **username_1**: The code for that paper (among others) is located [here](https://github.com/google-research/vision_transformer). If someone is familiar with Tensorflow and Python then they should be able to figure out the answer to your question (note that you can also answer it yourself if you read through the code and discover it).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-04. Score: 6 | Accepted answer
  
  Yes, they can handle sequences with arbitrary length sequence, but with some remarks.
  
  In the paper [Training data-efficient image transformers & distillation through attention](https://arxiv.org/abs/2012.12877v2) authors train models in the resolution 224x224 (1 + 14x14 tokens) and then finetune to the 384x384 (1 + 28x28 tokens).
  
  Weights to produce queries, keys, values, as well as feedforward layers, operate only on a single token and are agnostic to the sequence length.
  
  However, the size of the sequence is required in the positional embeddings, where one has a specific weight for each location of the token.
  
  In order to make this construction work for inputs of other sizes, one needs to transform this positional embedding in a certain way. **Bicubic interpolation** of positional embeddings, used in DeiT, works pretty well.
  One can use simpler **Bilinear** or **Nearest** interpolation - but it seems like this harms accuracy.
  
  In my own experience, when I took the DeiT-tiny with base accuracy of **72.2%** after several epochs of finetuning gave accuracy of **77.0%**.
  
  
  ---
  
  