<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="model = AutoModelForSequenceClassification.from_pretrained(&quot;bert-base-uncased&quot;)"><pre class="notranslate"><code class="notranslate">model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
</code></pre></div>
<p dir="auto">returns this warning message:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference."><pre class="notranslate"><code class="notranslate">Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
</code></pre></div>
<p dir="auto">This just started popping up with v.3 so I'm not sure what is the recommended action to take here.  Please advise if you can.  Basically, any of my code using the <code class="notranslate">AutoModelFor&lt;X&gt;</code> is throwing up this warning now.</p>
<p dir="auto">Thanks.</p>
<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.17.0</li>
<li>Platform: Linux-5.10.0-051000-generic-x86_64-with-glibc2.10</li>
<li>Python version: 3.8.5</li>
<li>PyTorch version (GPU?): 1.10.2+cu113 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: Yes (the same on CPU)</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Narsil/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Narsil">@Narsil</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/SaulLu/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/SaulLu">@SaulLu</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): BERT (<code class="notranslate">bert-large-cased</code>)</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="encoded = tokenizer(&quot;This thing costs £4.56&quot;)
decoded = tokenizer.decode(encoded[&quot;input_ids&quot;], clean_up_tokenization_spaces=True)
print (decoded)"><pre class="notranslate"><span class="pl-s1">encoded</span> <span class="pl-c1">=</span> <span class="pl-en">tokenizer</span>(<span class="pl-s">"This thing costs £4.56"</span>)
<span class="pl-s1">decoded</span> <span class="pl-c1">=</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>(<span class="pl-s1">encoded</span>[<span class="pl-s">"input_ids"</span>], <span class="pl-s1">clean_up_tokenization_spaces</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-en">print</span> (<span class="pl-s1">decoded</span>)</pre></div>
<p dir="auto">Real output: <code class="notranslate">[CLS] This thing costs £4. 56 [SEP]</code></p>
<p dir="auto">I tried it also with NER pipelines and other text inputs.<br>
Additional example: got <code class="notranslate">[CLS] ( including once - a - week tapping ) [SEP]</code> instead of <code class="notranslate">[CLS] (including once-a-week tapping) [SEP]</code></p>
<h2 dir="auto">Expected behavior</h2>

<p dir="auto">Expected output: <code class="notranslate">[CLS] This thing costs £4.56 [SEP]</code>.</p>
<p dir="auto">I expected the tokenizer to cleanup all the spaces introduced. Is there any different way to do so? Am I missing some trivial parameter?</p>