<h1 dir="auto">🚀 Feature request</h1>
<p dir="auto">I was just reading how <code class="notranslate">make style</code> can be automated with pre-commit hooks. Noticing how often I run and even more often forget to run <code class="notranslate">make style</code> before committing, perhaps others are in the same boat - and therefore I thought to propose to the dev community to (mostly) automate this process. The only cons is that each dev will still have to run <code class="notranslate">pre-commit install</code> once after cloning the project. This is a security feature of git, so it won't run anything automatically unless you take action to enable such thing.</p>
<p dir="auto">If I understand it correctly, if an individual dev doesn't run <code class="notranslate">pre-commit install</code> inside the repo, things are just as normal as they are now. That dev will just run <code class="notranslate">make style</code> manually. i.e. the proposed feature is optional for those who want it.</p>
<p dir="auto">I read about it <a href="https://www.mattlayman.com/blog/2018/python-code-black/" rel="nofollow">here</a>, please scroll down to the section: "Black as a Git pre-commit hook". And it links to the whole detailed website: <a href="https://pre-commit.com/" rel="nofollow">https://pre-commit.com/</a></p>
<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.19.2</li>
<li>Platform: Linux-5.4.0-72-generic-x86_64-with-glibc2.29</li>
<li>Python version: 3.8.10</li>
<li>Huggingface_hub version: 0.7.0</li>
<li>PyTorch version (GPU?): 1.11.0+cu113 (False)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: </li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('facebook/opt-350m')
tok = AutoTokenizer.from_pretrained('facebook/opt-350m', use_fast=False)
    
print(model.config.vocab_size)        # 50272
print(tok.vocab_size)                      # 50265"><pre class="notranslate"><code class="notranslate">from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('facebook/opt-350m')
tok = AutoTokenizer.from_pretrained('facebook/opt-350m', use_fast=False)
    
print(model.config.vocab_size)        # 50272
print(tok.vocab_size)                      # 50265
</code></pre></div>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">Hello,<br>
I'm not sure whether this is a bug or if I am missing something.<br>
In the reproduction script above, the model has a bigger vocabulary than the tokenizer. In my project, the LM produces the token <code class="notranslate">50272</code>, which the tokenizer doesn't know and thus the decode() function fails.<br>
(I use my own text generation script, so is it by any chance that the model is not supposed to output the last 7 tokens that the tokenizer doesn't know?)</p>
<p dir="auto">Best, David</p>