<p dir="auto">Hi, hope all is well :)</p>
<p dir="auto">It looks like DistilBertTokenizerFast doesn't take do_lower_case into account.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import DistilBertTokenizerFast, DistilBertTokenizer

PRE_TRAINED_MODEL_NAME = &quot;distilbert-base-uncased&quot; 
tokenizer_f = DistilBertTokenizerFast.from_pretrained(PRE_TRAINED_MODEL_NAME, do_lower_case=False)
tokenizer_s = DistilBertTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME, do_lower_case=False)   

sample = &quot;Hello, world. How are you?&quot;

tokens_f = tokenizer_f.tokenize(sample)
tokens_s = tokenizer_s.tokenize(sample)

print(tokens_f)
print(tokens_s)"><pre class="notranslate"><code class="notranslate">from transformers import DistilBertTokenizerFast, DistilBertTokenizer

PRE_TRAINED_MODEL_NAME = "distilbert-base-uncased" 
tokenizer_f = DistilBertTokenizerFast.from_pretrained(PRE_TRAINED_MODEL_NAME, do_lower_case=False)
tokenizer_s = DistilBertTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME, do_lower_case=False)   

sample = "Hello, world. How are you?"

tokens_f = tokenizer_f.tokenize(sample)
tokens_s = tokenizer_s.tokenize(sample)

print(tokens_f)
print(tokens_s)
</code></pre></div>
<p dir="auto">output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="['hello', ',', 'world', '.', 'how', 'are', 'you', '?']
['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']"><pre class="notranslate"><code class="notranslate">['hello', ',', 'world', '.', 'how', 'are', 'you', '?']
['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']
</code></pre></div>
<p dir="auto">expected:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']
['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']"><pre class="notranslate"><code class="notranslate">['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']
['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']
</code></pre></div>
<p dir="auto">packages:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
backcall==0.2.0
bleach==3.3.0
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
click==7.1.2
decorator==4.4.2
defusedxml==0.7.1
entrypoints==0.3
filelock==3.0.12
idna==2.10
ipykernel==5.5.0
ipython==7.21.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==2.11.3
joblib==1.0.1
jsonschema==3.2.0
jupyter-client==6.1.11
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
MarkupSafe==1.1.1
mistune==0.8.4
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.2
nest-asyncio==1.5.1
notebook==6.2.0
numpy==1.20.1
packaging==20.9
pandocfilters==1.4.3
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.9.0
prompt-toolkit==3.0.16
ptyprocess==0.7.0
pycparser==2.20
Pygments==2.8.1
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pyzmq==22.0.3
regex==2020.11.13
requests==2.25.1
sacremoses==0.0.43
Send2Trash==1.5.0
six==1.15.0
terminado==0.9.2
testpath==0.4.4
tokenizers==0.10.1
torch==1.8.0+cu111
tornado==6.1
tqdm==4.59.0
traitlets==5.0.5
transformers==4.3.3
typing-extensions==3.7.4.3
urllib3==1.26.3
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1"><pre class="notranslate"><code class="notranslate">argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
backcall==0.2.0
bleach==3.3.0
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
click==7.1.2
decorator==4.4.2
defusedxml==0.7.1
entrypoints==0.3
filelock==3.0.12
idna==2.10
ipykernel==5.5.0
ipython==7.21.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==2.11.3
joblib==1.0.1
jsonschema==3.2.0
jupyter-client==6.1.11
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
MarkupSafe==1.1.1
mistune==0.8.4
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.2
nest-asyncio==1.5.1
notebook==6.2.0
numpy==1.20.1
packaging==20.9
pandocfilters==1.4.3
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.9.0
prompt-toolkit==3.0.16
ptyprocess==0.7.0
pycparser==2.20
Pygments==2.8.1
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pyzmq==22.0.3
regex==2020.11.13
requests==2.25.1
sacremoses==0.0.43
Send2Trash==1.5.0
six==1.15.0
terminado==0.9.2
testpath==0.4.4
tokenizers==0.10.1
torch==1.8.0+cu111
tornado==6.1
tqdm==4.59.0
traitlets==5.0.5
transformers==4.3.3
typing-extensions==3.7.4.3
urllib3==1.26.3
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
</code></pre></div>
<p dir="auto">Python version:<br>
<code class="notranslate">Python 3.8.6</code></p>
<p dir="auto">System:<br>
PopOS 20, happy to provide more info on system specs such as hardware if needed</p>
<h3 dir="auto">System Info</h3>
<p dir="auto">2023-05-24 23:09:53.575434: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT<br>
WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/transformers/commands/env.py:63: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.<br>
Instructions for updating:<br>
Use <code class="notranslate">tf.config.list_physical_devices('GPU')</code> instead.<br>
2023-05-24 23:10:05.261610: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.</p>
<p dir="auto">Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.</p>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.29.2</li>
<li>Platform: Linux-5.15.107+-x86_64-with-glibc2.31</li>
<li>Python version: 3.10.11</li>
<li>Huggingface_hub version: 0.14.1</li>
<li>Safetensors version: not installed</li>
<li>PyTorch version (GPU?): 2.0.1+cu118 (True)</li>
<li>Tensorflow version (GPU?): 2.12.0 (True)</li>
<li>Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)</li>
<li>Jax version: 0.4.10</li>
<li>JaxLib version: 0.4.10</li>
<li>Using GPU in script?: </li>
<li>Using distributed or parallel set-up in script?: </li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ArthurZucker/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ArthurZucker">@ArthurZucker</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/younesbelkada/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/younesbelkada">@younesbelkada</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">I was trying to use the new shiny mpt model from the huggingface hub from a revision :</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
import torch
import transformers
import accelerate

model_name = 'mosaicml/mpt-7b'

model = AutoModelForCausalLM.from_pretrained(model_name, 
                                            trust_remote_code=True,
                                            revision=&quot;refs/pr/23&quot;,
                                            device_map=&quot;auto&quot;
                                          )"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoConfig</span>, <span class="pl-v">AutoModelForCausalLM</span>, <span class="pl-v">AutoTokenizer</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">import</span> <span class="pl-s1">transformers</span>
<span class="pl-k">import</span> <span class="pl-s1">accelerate</span>

<span class="pl-s1">model_name</span> <span class="pl-c1">=</span> <span class="pl-s">'mosaicml/mpt-7b'</span>

<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">AutoModelForCausalLM</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">model_name</span>, 
                                            <span class="pl-s1">trust_remote_code</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
                                            <span class="pl-s1">revision</span><span class="pl-c1">=</span><span class="pl-s">"refs/pr/23"</span>,
                                            <span class="pl-s1">device_map</span><span class="pl-c1">=</span><span class="pl-s">"auto"</span>
                                          )</pre></div>
<p dir="auto">But I stumble on this error after the using the above code :<br>
<code class="notranslate">ValueError: MPTForCausalLM does not support </code>device_map='auto'<code class="notranslate"> yet.</code></p>
<p dir="auto">The "auto" was indeed not supported in the main branch but we add a correction in the PR branch (so the argument revision="refs/pr/23")<br>
I did some investigation and the model was indeed loading the main .py files :</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Downloading (…)main/modeling_mpt.py: 100%
17.4k/17.4k [00:00&lt;00:00, 1.12MB/s]
Downloading (…)in/param_init_fns.py: 100%
12.6k/12.6k [00:00&lt;00:00, 971kB/s]
Downloading (…)resolve/main/norm.py: 100%
2.56k/2.56k [00:00&lt;00:00, 131kB/s]
A new version of the following files was downloaded from https://huggingface.co/mosaicml/mpt-7b:
- norm.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/mosaicml/mpt-7b:
- param_init_fns.py
- norm.py"><pre class="notranslate"><code class="notranslate">Downloading (…)main/modeling_mpt.py: 100%
17.4k/17.4k [00:00&lt;00:00, 1.12MB/s]
Downloading (…)in/param_init_fns.py: 100%
12.6k/12.6k [00:00&lt;00:00, 971kB/s]
Downloading (…)resolve/main/norm.py: 100%
2.56k/2.56k [00:00&lt;00:00, 131kB/s]
A new version of the following files was downloaded from https://huggingface.co/mosaicml/mpt-7b:
- norm.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/mosaicml/mpt-7b:
- param_init_fns.py
- norm.py
</code></pre></div>
<p dir="auto">You can see the main/ here. I did manually check the modeling_mpt.py file it didn't have the PR changes.</p>
<p dir="auto">So I did try to find where the bug where inside the transformers package ... (first time looking at the code).<br>
I am a bit surprised !<br>
Basicly the code rewrite the config values after having read it (it adds the information about the repo ids (in add_model_info_to_auto_map in generic.py in utils/ from the transformers package) something that seems normal.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&quot;auto_map&quot;: {
    &quot;AutoConfig&quot;: &quot;mosaicml/mpt-7b--configuration_mpt.MPTConfig&quot;,
    &quot;AutoModelForCausalLM&quot;: &quot;mosaicml/mpt-7b--modeling_mpt.MPTForCausalLM&quot;
  }"><pre class="notranslate"><code class="notranslate">"auto_map": {
    "AutoConfig": "mosaicml/mpt-7b--configuration_mpt.MPTConfig",
    "AutoModelForCausalLM": "mosaicml/mpt-7b--modeling_mpt.MPTForCausalLM"
  }
</code></pre></div>
<p dir="auto">It notably add the "--" string.<br>
then in get_class_from_dynamic_module (in dynamic_module_utils.py) it has :</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="if &quot;--&quot; in class_reference:
        repo_id, class_reference = class_reference.split(&quot;--&quot;)
        # Invalidate revision since it's not relevant for this repo
        revision = &quot;main&quot;"><pre class="notranslate"><code class="notranslate">if "--" in class_reference:
        repo_id, class_reference = class_reference.split("--")
        # Invalidate revision since it's not relevant for this repo
        revision = "main"
</code></pre></div>
<p dir="auto">So the revision become "main" and from here we are done.<br>
I suppose if i do a PR removing the revision overide some people will not be happy ?</p>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">The expected behaviour is to load the file from the PR branch. (not the main/)</p>