Answers.Body: <p>If you're using a library such as <a href="https://github.com/google/trax" rel="nofollow noreferrer">Trax</a> which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt <code>trax.data.inputs.add_loss_weights()</code> function and provide a value for the <code>id_to_mask</code> parameter.</p>
  <p>Example Usage:</p>
  <pre><code>train_generator = trax.data.inputs.add_loss_weights(
  data_generator(batch_size, x_train, y_train,vocab['&lt;PAD&gt;'], True),
  id_to_mask=vocab['&lt;PAD&gt;'])
  </code></pre>
  <p>Here are some resources for building Transformers in Trax:</p>
  <ul>
  <li><p><a href="https://github.com/SauravMaheshkar/Trax-Examples/blob/main/NLP/NER%20using%20Reformer.ipynb" rel="nofollow noreferrer">Named Entity Recognition with Reformer</a> (Masking Implemented) and the associated <a href="https://sauravmaheshkar.github.io/fastpages/transformers/jupyter/nlp/2020/11/29/_11_19_NER_Using_Reformer.html" rel="nofollow noreferrer">blog post</a> (contains an in-depth explanation of NER and Transformers)</p>
  </li>
  <li><p><a href="https://github.com/SauravMaheshkar/Trax-Examples/blob/main/NLP/Deep%20N-Gram.ipynb" rel="nofollow noreferrer">Deep N-Gram Models</a> (without masking but I recommend you go through these first to get a hang of the library)</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-29T13:36:47.193Z
Answers.Text: The associated blog post link gives me a 404. Is there an up to date version?
Answers.UserDisplayName: Alan Buxton
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-29T14:45:26.077Z
Answers.Text: Here's the updated [link](https://sauravmaheshkar.github.io/blog/blog/NER-Using-Reformer/). Sorry I moved my blog from fastpages 
Answers.UserDisplayName: Saurav Maheshkar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-25T19:59:13.870Z
Answers.Text: @SauravMaheshkar It seems the link is still not working.
Answers.UserDisplayName: Shayan Shafiq
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-26T08:09:27.850Z
Answers.Text: I keep changing blog platforms a lot so the links keep changing. I'll make it a point to add a permalink next time. For now, the link is [this](https://sauravmaheshkar.github.io/blog/projects/NER-Using-Reformer/).  I apologize but the Markdown parser is still not fixed so the latex is going to be hard to read. If you necessarily need the latex rendered visit this [alternative link](https://sauravvmaheshkar.gitbook.io/saurav-maheshkar/projects/named-entity-recognition-using-reformers/named-entity-recognition-a-brief-overview)
Answers.UserDisplayName: Saurav Maheshkar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-01T14:59:08.630Z
Answers.LastActivityDate: 2020-12-01T14:59:08.630Z
Answers.UserDisplayName: Saurav Maheshkar
Body: <p>When it comes to using Transformers for image captioning is there any reason to use masking?</p>
  <p>I currently have a resnet101 encoder and am trying to use the features as the input for a transformer model in order to generate a caption for the image, is there any need to use masking? and what would I mask if I did need to?</p>
  <p>Any help would be much appreciated</p>
  <p>Thanks in advance.</p>
  
Tags: <convolutional-neural-networks><natural-language-processing><python><pytorch><transformer>
Title: How to implement or avoid masking for transformer?
UserDisplayName: Gibbo0789
ThreadText: # How to implement or avoid masking for transformer?
  
  (Asked by: username_0 on 2020-09-22)
  
  When it comes to using Transformers for image captioning is there any reason to use masking?
  
  I currently have a resnet101 encoder and am trying to use the features as the input for a transformer model in order to generate a caption for the image, is there any need to use masking? and what would I mask if I did need to?
  
  Any help would be much appreciated
  
  Thanks in advance.
  
  **Tags:** convolutional-neural-networks, natural-language-processing, python, pytorch, transformer
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-01. Score: 2
  
  If you're using a library such as [Trax](https://github.com/google/trax) which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt `trax.data.inputs.add_loss_weights()` function and provide a value for the `id_to_mask` parameter.
  
  Example Usage:
  
  ```
  train_generator = trax.data.inputs.add_loss_weights(
  data_generator(batch_size, x_train, y_train,vocab['<PAD>'], True),
  id_to_mask=vocab['<PAD>'])
  
  ```
  
  Here are some resources for building Transformers in Trax:
  
  * [Named Entity Recognition with Reformer](https://github.com/username_1/Trax-Examples/blob/main/NLP/NER%20using%20Reformer.ipynb) (Masking Implemented) and the associated [blog post](https://username_1.github.io/fastpages/transformers/jupyter/nlp/2020/11/29/_11_19_NER_Using_Reformer.html) (contains an in-depth explanation of NER and Transformers)
  * [Deep N-Gram Models](https://github.com/username_1/Trax-Examples/blob/main/NLP/Deep%20N-Gram.ipynb) (without masking but I recommend you go through these first to get a hang of the library)
  
  > **Comments:**
  > **username_2**: The associated blog post link gives me a 404. Is there an up to date version?  
  > **username_1**: Here's the updated [link](https://username_1.github.io/blog/blog/NER-Using-Reformer/). Sorry I moved my blog from fastpages   
  > **username_3**: @username_1 It seems the link is still not working.  
  > **username_1**: I keep changing blog platforms a lot so the links keep changing. I'll make it a point to add a permalink next time. For now, the link is [this](https://username_1.github.io/blog/projects/NER-Using-Reformer/).  I apologize but the Markdown parser is still not fixed so the latex is going to be hard to read. If you necessarily need the latex rendered visit this [alternative link](https://sauravvmaheshkar.gitbook.io/username_1/projects/named-entity-recognition-using-reformers/named-entity-recognition-a-brief-overview)  
  
  ---
  
  