<p>If you're using a library such as <a href="https://github.com/google/trax" rel="nofollow noreferrer">Trax</a> which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt <code>trax.data.inputs.add_loss_weights()</code> function and provide a value for the <code>id_to_mask</code> parameter.</p>
<p>Example Usage:</p>
<pre><code>train_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_train, y_train,vocab['&lt;PAD&gt;'], True),
id_to_mask=vocab['&lt;PAD&gt;'])
</code></pre>
<p>Here are some resources for building Transformers in Trax:</p>
<ul>
<li><p><a href="https://github.com/SauravMaheshkar/Trax-Examples/blob/main/NLP/NER%20using%20Reformer.ipynb" rel="nofollow noreferrer">Named Entity Recognition with Reformer</a> (Masking Implemented) and the associated <a href="https://sauravmaheshkar.github.io/fastpages/transformers/jupyter/nlp/2020/11/29/_11_19_NER_Using_Reformer.html" rel="nofollow noreferrer">blog post</a> (contains an in-depth explanation of NER and Transformers)</p>
</li>
<li><p><a href="https://github.com/SauravMaheshkar/Trax-Examples/blob/main/NLP/Deep%20N-Gram.ipynb" rel="nofollow noreferrer">Deep N-Gram Models</a> (without masking but I recommend you go through these first to get a hang of the library)</p>
</li>
</ul>

CC BY-SA 4.0
2020-12-29T13:36:47.193Z
The associated blog post link gives me a 404. Is there an up to date version?
Alan Buxton
CC BY-SA 4.0
2020-12-29T14:45:26.077Z
Here's the updated [link](https://sauravmaheshkar.github.io/blog/blog/NER-Using-Reformer/). Sorry I moved my blog from fastpages 
Saurav Maheshkar
CC BY-SA 4.0
2021-04-25T19:59:13.870Z
@SauravMaheshkar It seems the link is still not working.
Shayan Shafiq
CC BY-SA 4.0
2021-04-26T08:09:27.850Z
I keep changing blog platforms a lot so the links keep changing. I'll make it a point to add a permalink next time. For now, the link is [this](https://sauravmaheshkar.github.io/blog/projects/NER-Using-Reformer/).  I apologize but the Markdown parser is still not fixed so the latex is going to be hard to read. If you necessarily need the latex rendered visit this [alternative link](https://sauravvmaheshkar.gitbook.io/saurav-maheshkar/projects/named-entity-recognition-using-reformers/named-entity-recognition-a-brief-overview)
Saurav Maheshkar
CC BY-SA 4.0
2020-12-01T14:59:08.630Z
2020-12-01T14:59:08.630Z
Saurav Maheshkar
<p>When it comes to using Transformers for image captioning is there any reason to use masking?</p>
<p>I currently have a resnet101 encoder and am trying to use the features as the input for a transformer model in order to generate a caption for the image, is there any need to use masking? and what would I mask if I did need to?</p>
<p>Any help would be much appreciated</p>
<p>Thanks in advance.</p>

<convolutional-neural-networks><natural-language-processing><python><pytorch><transformer>
How to implement or avoid masking for transformer?
Gibbo0789
# How to implement or avoid masking for transformer?

(Asked by: username_0 on 2020-09-22)

When it comes to using Transformers for image captioning is there any reason to use masking?

I currently have a resnet101 encoder and am trying to use the features as the input for a transformer model in order to generate a caption for the image, is there any need to use masking? and what would I mask if I did need to?

Any help would be much appreciated

Thanks in advance.

**Tags:** convolutional-neural-networks, natural-language-processing, python, pytorch, transformer


---

## 1 answers

### Answer by username_1 on 2020-12-01. Score: 2

If you're using a library such as Trax which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt trax.data.inputs.add_loss_weights() function and provide a value for the id_to_mask parameter.
Example Usage:
train_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_train, y_train,vocab[''], True),
id_to_mask=vocab[''])

Here are some resources for building Transformers in Trax:

Named Entity Recognition with Reformer (Masking Implemented) and the associated blog post (contains an in-depth explanation of NER and Transformers)

Deep N-Gram Models (without masking but I recommend you go through these first to get a hang of the library)

> **Comments:**
> **username_2**: The associated blog post link gives me a 404. Is there an up to date version?  
> **username_1**: Here's the updated [link](https://sauravmaheshkar.github.io/blog/blog/NER-Using-Reformer/). Sorry I moved my blog from fastpages   
> **username_3**: @SauravMaheshkar It seems the link is still not working.  
> **username_1**: I keep changing blog platforms a lot so the links keep changing. I'll make it a point to add a permalink next time. For now, the link is [this](https://sauravmaheshkar.github.io/blog/projects/NER-Using-Reformer/).  I apologize but the Markdown parser is still not fixed so the latex is going to be hard to read. If you necessarily need the latex rendered visit this [alternative link](https://sauravvmaheshkar.gitbook.io/saurav-maheshkar/projects/named-entity-recognition-using-reformers/named-entity-recognition-a-brief-overview)  

---

