<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.2.2</li>
<li>Platform: Linux-4.15.0-45-generic-x86_64-with-glibc2.10</li>
<li>Python version: 3.8.3</li>
<li>PyTorch version (GPU?): 1.6.0 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Using GPU in script?: No</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>
<h3 dir="auto">Who can help</h3>

<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...):</p>
<p dir="auto">The problem arises when using:</p>
<ul dir="auto">
<li>my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul dir="auto">
<li>my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>The code
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(&quot;bert-base-uncased&quot;)
words = ['(cid:3)', '하셨습니까', '하다']
tokenizer.batch_encode_plus(
    [words],
    max_length=512,
    truncation=True,
    padding=True,
    is_split_into_words=True,
    return_offsets_mapping=True,
    return_special_tokens_mask=True,
    return_tensors=&quot;pt&quot;,
)"><pre class="notranslate"><code class="notranslate">from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
words = ['(cid:3)', '하셨습니까', '하다']
tokenizer.batch_encode_plus(
    [words],
    max_length=512,
    truncation=True,
    padding=True,
    is_split_into_words=True,
    return_offsets_mapping=True,
    return_special_tokens_mask=True,
    return_tensors="pt",
)
</code></pre></div>
</li>
<li>The <code class="notranslate">offset_mapping</code> in the output is
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="tensor([[[0, 0],        # [CLS]
         [0, 1],        # for '('
         [1, 4],        # for 'cid'
         [4, 5],        # for ':'
         [5, 6],        # for '3'
         [6, 7],        # for ')'
         [0, 5],        # for '하셨습니까'
         [0, 1],        # for '하'
         [0, 1],        # for '하'
         [1, 2],        # for '다'
         [1, 2],        # for '다'
         [0, 0]]])"><pre class="notranslate"><span class="pl-en">tensor</span>([[[<span class="pl-c1">0</span>, <span class="pl-c1">0</span>],        <span class="pl-c"># [CLS]</span>
         [<span class="pl-c1">0</span>, <span class="pl-c1">1</span>],        <span class="pl-c"># for '('</span>
         [<span class="pl-c1">1</span>, <span class="pl-c1">4</span>],        <span class="pl-c"># for 'cid'</span>
         [<span class="pl-c1">4</span>, <span class="pl-c1">5</span>],        <span class="pl-c"># for ':'</span>
         [<span class="pl-c1">5</span>, <span class="pl-c1">6</span>],        <span class="pl-c"># for '3'</span>
         [<span class="pl-c1">6</span>, <span class="pl-c1">7</span>],        <span class="pl-c"># for ')'</span>
         [<span class="pl-c1">0</span>, <span class="pl-c1">5</span>],        <span class="pl-c"># for '하셨습니까'</span>
         [<span class="pl-c1">0</span>, <span class="pl-c1">1</span>],        <span class="pl-c"># for '하'</span>
         [<span class="pl-c1">0</span>, <span class="pl-c1">1</span>],        <span class="pl-c"># for '하'</span>
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>],        <span class="pl-c"># for '다'</span>
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>],        <span class="pl-c"># for '다'</span>
         [<span class="pl-c1">0</span>, <span class="pl-c1">0</span>]]])</pre></div>
</li>
<li>As you could find, it generates four tokens for <code class="notranslate">하다</code>. The output is correct according to Byte pair encoding. However, it generates duplicated <code class="notranslate">[0,1]</code> and <code class="notranslate">[1,2]</code>s, which changes the structure of the outputs (for regular tokens, it can only have one <code class="notranslate">[0,x]</code>, which can be used to project the encoded tokens back to their original positions). Therefore, we need extra indicators for positions where Byte-pair encoding is used.</li>
</ol>

<h2 dir="auto">Expected behavior</h2>

<ol dir="auto">
<li>An additional output showing the mapping for input_ids -&gt; original_token_ids . In this case, it should be something like:
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[0, 1, 1, 1, 1, 1, 2, 3, 3, 3, 3 0]"><pre class="notranslate"><code class="notranslate">[0, 1, 1, 1, 1, 1, 2, 3, 3, 3, 3 0]
</code></pre></div>
</li>
</ol>
<p dir="auto">Therefore, we could use this map to figure out byte code embedding is used for the 3rd token.</p>
<p dir="auto">Updated - <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/n1t0/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/n1t0">@n1t0</a></p>
<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.12.2</li>
<li>Platform: Linux-5.4.0-87-generic-x86_64-with-glibc2.10</li>
<li>Python version: 3.8.8</li>
<li>PyTorch version (GPU?): 1.9.1+cu111 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: Yes</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): BertLMHeadModel</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Instantiate a BertLMHeadModel. The model contains, among others, a linear layer under the path <code class="notranslate">cls.predictions.decoder </code>. Specifically, it contains 2 parameters: <code class="notranslate">cls.predictions.decoder.weight</code> and <code class="notranslate">cls.predictions.decoder.bias</code>.</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(&quot;bert-base-uncased&quot;, is_decoder=True)
print(model.cls.predictions.decoder)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoModelForCausalLM</span>
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">AutoModelForCausalLM</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"bert-base-uncased"</span>, <span class="pl-s1">is_decoder</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-en">print</span>(<span class="pl-s1">model</span>.<span class="pl-s1">cls</span>.<span class="pl-s1">predictions</span>.<span class="pl-s1">decoder</span>)</pre></div>
<ol start="2" dir="auto">
<li>Get a list of model parameters.</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="parameter_keys = list(dict(model.named_parameters()).keys())"><pre class="notranslate"><span class="pl-s1">parameter_keys</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-en">dict</span>(<span class="pl-s1">model</span>.<span class="pl-en">named_parameters</span>()).<span class="pl-en">keys</span>())</pre></div>
<ol start="3" dir="auto">
<li>The decoder parameters are missing.</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="print(&quot;cls.predictions.decoder.weight&quot; in parameter_keys)
print(&quot;cls.predictions.decoder.bias&quot; in parameter_keys)"><pre class="notranslate"><span class="pl-en">print</span>(<span class="pl-s">"cls.predictions.decoder.weight"</span> <span class="pl-c1">in</span> <span class="pl-s1">parameter_keys</span>)
<span class="pl-en">print</span>(<span class="pl-s">"cls.predictions.decoder.bias"</span> <span class="pl-c1">in</span> <span class="pl-s1">parameter_keys</span>)</pre></div>
<ol start="4" dir="auto">
<li>Note that the parameters appear under the cls module:</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="print(list(dict(model.cls.named_parameters()).keys()))"><pre class="notranslate"><span class="pl-en">print</span>(<span class="pl-en">list</span>(<span class="pl-en">dict</span>(<span class="pl-s1">model</span>.<span class="pl-s1">cls</span>.<span class="pl-en">named_parameters</span>()).<span class="pl-en">keys</span>()))</pre></div>
<ol start="5" dir="auto">
<li>Note that the <code class="notranslate">cls.decoder</code> module appears to be registerred.</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="print(list(dict(model.named_modules()).keys()))"><pre class="notranslate"><span class="pl-en">print</span>(<span class="pl-en">list</span>(<span class="pl-en">dict</span>(<span class="pl-s1">model</span>.<span class="pl-en">named_modules</span>()).<span class="pl-en">keys</span>()))</pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Decoder parameters should be incldued in the model parameters.</p>