<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.1.1</li>
<li>Platform: Linux-3.10.0-693.5.2.el7.x86_64-x86_64-with-centos-7.4.1708-Core</li>
<li>Python version: 3.7.9</li>
<li>PyTorch version (GPU?): 1.7.1 (False)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Using GPU in script?: </li>
<li>Using distributed or parallel set-up in script?: </li>
</ul>
<h3 dir="auto">Who can help</h3>

<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...):bert-base-uncased</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)<br>
i am fine-tuning a text_claasifiction on dbpedia_14.and i followed this colab <a href="https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb#scrollTo=TlqNaB8jIrJW" rel="nofollow">https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb#scrollTo=TlqNaB8jIrJW</a></li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)<br>
datset:dbpedia_14</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">error<br>
<code class="notranslate">File "train.py", line 69, in &lt;module&gt; trainer.train() File "/home/pliu3/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/transformers/trainer.py", line 784, in train for step, inputs in enumerate(epoch_iterator): File "/home/pliu3/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__ data = self._next_data() File "/home/pliu3/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data data = self._dataset_fetcher.fetch(index)  # may raise StopIteration File "/home/pliu3/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/pliu3/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in &lt;listcomp&gt; data = [self.dataset[idx] for idx in possibly_batched_index] KeyError: 2</code></p>
<p dir="auto">code</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="dataset_name = 'sem_eval_2014_task_1'

num_labels_size = 3

batch_size = 4

model_checkpoint = 'bert-base-uncased'

number_train_epoch = 5



def tokenize(batch):
    return tokenizer(batch['premise'], batch['hypothesis'],  truncation=True, )


def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='micro')
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }


model = BertForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels_size)
tokenizer = BertTokenizerFast.from_pretrained(model_checkpoint, use_fast=True)

train_dataset = load_dataset(dataset_name, split='train')
test_dataset = load_dataset(dataset_name, split='test')

train_encoded_dataset = train_dataset.map(tokenize, batched=True)
test_encoded_dataset = test_dataset.map(tokenize, batched=True)

args = TrainingArguments(
    output_dir='./results',
    evaluation_strategy=&quot;epoch&quot;,
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=number_train_epoch,
    weight_decay=0.01,
    do_predict=True
)
trainer = Trainer(
    model=model,
    args=args,
    compute_metrics=compute_metrics,
    train_dataset=train_encoded_dataset,
    eval_dataset=test_encoded_dataset,
    tokenizer=tokenizer
)

trainer.train()
trainer.evaluate()"><pre class="notranslate"><span class="pl-s1">dataset_name</span> <span class="pl-c1">=</span> <span class="pl-s">'sem_eval_2014_task_1'</span>

<span class="pl-s1">num_labels_size</span> <span class="pl-c1">=</span> <span class="pl-c1">3</span>

<span class="pl-s1">batch_size</span> <span class="pl-c1">=</span> <span class="pl-c1">4</span>

<span class="pl-s1">model_checkpoint</span> <span class="pl-c1">=</span> <span class="pl-s">'bert-base-uncased'</span>

<span class="pl-s1">number_train_epoch</span> <span class="pl-c1">=</span> <span class="pl-c1">5</span>



<span class="pl-k">def</span> <span class="pl-en">tokenize</span>(<span class="pl-s1">batch</span>):
    <span class="pl-k">return</span> <span class="pl-en">tokenizer</span>(<span class="pl-s1">batch</span>[<span class="pl-s">'premise'</span>], <span class="pl-s1">batch</span>[<span class="pl-s">'hypothesis'</span>],  <span class="pl-s1">truncation</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, )


<span class="pl-k">def</span> <span class="pl-en">compute_metrics</span>(<span class="pl-s1">pred</span>):
    <span class="pl-s1">labels</span> <span class="pl-c1">=</span> <span class="pl-s1">pred</span>.<span class="pl-s1">label_ids</span>
    <span class="pl-s1">preds</span> <span class="pl-c1">=</span> <span class="pl-s1">pred</span>.<span class="pl-s1">predictions</span>.<span class="pl-en">argmax</span>(<span class="pl-c1">-</span><span class="pl-c1">1</span>)
    <span class="pl-s1">precision</span>, <span class="pl-s1">recall</span>, <span class="pl-s1">f1</span>, <span class="pl-s1">_</span> <span class="pl-c1">=</span> <span class="pl-en">precision_recall_fscore_support</span>(<span class="pl-s1">labels</span>, <span class="pl-s1">preds</span>, <span class="pl-s1">average</span><span class="pl-c1">=</span><span class="pl-s">'micro'</span>)
    <span class="pl-s1">acc</span> <span class="pl-c1">=</span> <span class="pl-en">accuracy_score</span>(<span class="pl-s1">labels</span>, <span class="pl-s1">preds</span>)
    <span class="pl-k">return</span> {
        <span class="pl-s">'accuracy'</span>: <span class="pl-s1">acc</span>,
        <span class="pl-s">'f1'</span>: <span class="pl-s1">f1</span>,
        <span class="pl-s">'precision'</span>: <span class="pl-s1">precision</span>,
        <span class="pl-s">'recall'</span>: <span class="pl-s1">recall</span>
    }


<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">BertForSequenceClassification</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">model_checkpoint</span>, <span class="pl-s1">num_labels</span><span class="pl-c1">=</span><span class="pl-s1">num_labels_size</span>)
<span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">BertTokenizerFast</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s1">model_checkpoint</span>, <span class="pl-s1">use_fast</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)

<span class="pl-s1">train_dataset</span> <span class="pl-c1">=</span> <span class="pl-en">load_dataset</span>(<span class="pl-s1">dataset_name</span>, <span class="pl-s1">split</span><span class="pl-c1">=</span><span class="pl-s">'train'</span>)
<span class="pl-s1">test_dataset</span> <span class="pl-c1">=</span> <span class="pl-en">load_dataset</span>(<span class="pl-s1">dataset_name</span>, <span class="pl-s1">split</span><span class="pl-c1">=</span><span class="pl-s">'test'</span>)

<span class="pl-s1">train_encoded_dataset</span> <span class="pl-c1">=</span> <span class="pl-s1">train_dataset</span>.<span class="pl-en">map</span>(<span class="pl-s1">tokenize</span>, <span class="pl-s1">batched</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">test_encoded_dataset</span> <span class="pl-c1">=</span> <span class="pl-s1">test_dataset</span>.<span class="pl-en">map</span>(<span class="pl-s1">tokenize</span>, <span class="pl-s1">batched</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)

<span class="pl-s1">args</span> <span class="pl-c1">=</span> <span class="pl-v">TrainingArguments</span>(
    <span class="pl-s1">output_dir</span><span class="pl-c1">=</span><span class="pl-s">'./results'</span>,
    <span class="pl-s1">evaluation_strategy</span><span class="pl-c1">=</span><span class="pl-s">"epoch"</span>,
    <span class="pl-s1">learning_rate</span><span class="pl-c1">=</span><span class="pl-c1">2e-5</span>,
    <span class="pl-s1">per_device_train_batch_size</span><span class="pl-c1">=</span><span class="pl-s1">batch_size</span>,
    <span class="pl-s1">per_device_eval_batch_size</span><span class="pl-c1">=</span><span class="pl-s1">batch_size</span>,
    <span class="pl-s1">num_train_epochs</span><span class="pl-c1">=</span><span class="pl-s1">number_train_epoch</span>,
    <span class="pl-s1">weight_decay</span><span class="pl-c1">=</span><span class="pl-c1">0.01</span>,
    <span class="pl-s1">do_predict</span><span class="pl-c1">=</span><span class="pl-c1">True</span>
)
<span class="pl-s1">trainer</span> <span class="pl-c1">=</span> <span class="pl-v">Trainer</span>(
    <span class="pl-s1">model</span><span class="pl-c1">=</span><span class="pl-s1">model</span>,
    <span class="pl-s1">args</span><span class="pl-c1">=</span><span class="pl-s1">args</span>,
    <span class="pl-s1">compute_metrics</span><span class="pl-c1">=</span><span class="pl-s1">compute_metrics</span>,
    <span class="pl-s1">train_dataset</span><span class="pl-c1">=</span><span class="pl-s1">train_encoded_dataset</span>,
    <span class="pl-s1">eval_dataset</span><span class="pl-c1">=</span><span class="pl-s1">test_encoded_dataset</span>,
    <span class="pl-s1">tokenizer</span><span class="pl-c1">=</span><span class="pl-s1">tokenizer</span>
)

<span class="pl-s1">trainer</span>.<span class="pl-en">train</span>()
<span class="pl-s1">trainer</span>.<span class="pl-en">evaluate</span>()</pre></div>
<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature request</h1>
<p dir="auto"><code class="notranslate">transformers.BertTokenizer.from_pretrained()</code> calls <code class="notranslate">get_fast_tokenizer_file()</code> which downloads a file from the HuggingFace server but never adds it to the cache.</p>
<p dir="auto">Would be useful to cache that file (in the same folder as the models) to make CI runs more robust.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">Try to avoid having issues such as the one below in our CI runs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E           requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/models/mrm8488/bert-small-finetuned-squadv2"><pre class="notranslate"><code class="notranslate">E           requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/models/mrm8488/bert-small-finetuned-squadv2
</code></pre></div>
<p dir="auto">This is a duplicate of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1085873504" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/14862" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/14862/hovercard" href="https://github.com/huggingface/transformers/issues/14862">#14862</a> which was closed without getting fixed and I don't seem able to re-open it.</p>