<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.10.0 (currently master)</li>
<li>Platform: TPU VM3.8 --  Ubuntu 20.04.2 LTS</li>
<li>Python version: 3.8.10</li>
<li>PyTorch version (GPU?): XLA - 1.8.1</li>
<li>Tensorflow version (GPU?): None</li>
<li>Using GPU in script?: None</li>
<li>Using distributed or parallel set-up in script?: Using <code class="notranslate">examples/pytorch/language-modeling/run_mlm_no_trainer.py</code> which is using Accelerator</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a>  <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patil-suraj/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patil-suraj">@patil-suraj</a></p>

<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...):</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">I have modified small things in  <code class="notranslate">examples/pytorch/language-modeling/run_mlm_no_trainer.py</code> and changes as follow (can be reached at <a href="https://github.com/akalieren/transformers-master">https://github.com/akalieren/transformers-master</a>)</p>
<ol dir="auto">
<li>Defined mp_fn to training script.</li>
<li>Added <code class="notranslate">streaming_data=True</code> to Dataset Class</li>
<li>Deleted <code class="notranslate">tpu_num_cores argument</code> from xla_spawn.py sys.args since it throw arrow.</li>
</ol>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> an official GLUE/SQUaD task: (give the name) Training MLM from scratch</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Clone modified script<br>
<code class="notranslate">git clone https://github.com/akalieren/transformers-master</code></li>
<li><code class="notranslate">export XRT_TPU_CONFIG="localservice;0;localhost:51011"</code></li>
<li>Install required libraries (I did not add extra installments to requirements.txt to highlight they are not stated in official example)</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install transformers-master
pip install .
pip install -r examples/pytorch/language-modeling/requirements.txt
pip install accelerate
pip install datasets[streaming]"><pre class="notranslate"><code class="notranslate">pip install transformers-master
pip install .
pip install -r examples/pytorch/language-modeling/requirements.txt
pip install accelerate
pip install datasets[streaming]
</code></pre></div>
<ol start="4" dir="auto">
<li>Run command</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python3 examples/pytorch/xla_spawn.py --num_cores 8 examples/pytorch/language-modeling/run_mlm_no_trainer.py   --model_type &quot;roberta&quot; --per_device_eval_batch_size 512 --per_device_train_batch_size 512 --max_train_steps 1000000 --preprocessing_num_workers 50 --pad_to_max_length  --tokenizer_name &quot;./tokenizers/Roberta/&quot;  --dataset_name='oscar'  --dataset_config_name='unshuffled_deduplicated_fr' --data_streaming=True --max_seq_length 512   --line_by_line=True"><pre class="notranslate"><code class="notranslate">python3 examples/pytorch/xla_spawn.py --num_cores 8 examples/pytorch/language-modeling/run_mlm_no_trainer.py   --model_type "roberta" --per_device_eval_batch_size 512 --per_device_train_batch_size 512 --max_train_steps 1000000 --preprocessing_num_workers 50 --pad_to_max_length  --tokenizer_name "./tokenizers/Roberta/"  --dataset_name='oscar'  --dataset_config_name='unshuffled_deduplicated_fr' --data_streaming=True --max_seq_length 512   --line_by_line=True
</code></pre></div>
<p dir="auto">Note: Without xla_spawn, Accelerator use only one cores. Thats why I changed, with 1 core it is running but slow</p>

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2021-07-26 00:30:54.355600: E tensorflow/core/framework/op_kernel.cc:1693] OpKernel ('op: &quot;TPURoundRobin&quot; device_type: &quot;CPU&quot;') for unknown op: TPURoundRobin
2021-07-26 00:30:54.355659: E tensorflow/core/framework/op_kernel.cc:1693] OpKernel ('op: &quot;TpuHandleToProtoKey&quot; device_type: &quot;CPU&quot;') for unknown op: TpuHandleToProtoKey
07/26/2021 00:31:13 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 0
Local process index: 0
Device: xla:1
Use FP16 precision: False

Downloading and preparing dataset oscar/unshuffled_deduplicated_tr (download: 9.68 GiB, generated: 26.43 GiB, post-processed: Unknown size, total: 36.10 GiB) to /home/akali/.cache/huggingface/datasets/oscar/unshuffled_deduplicated_tr/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
07/26/2021 00:31:20 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 1
Local process index: 1
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:20 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 5
Local process index: 5
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:20 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 7
Local process index: 7
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:20 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 6
Local process index: 6
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:21 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 2
Local process index: 2
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:21 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 4
Local process index: 4
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:23 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 3
Local process index: 3
Device: xla:0
Use FP16 precision: False

0 examples [00:00, ? examples/s]07/26/2021 00:31:44 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/657d72dc352d822d0496bb9f519cf0de87b87064d56024d9d1ac5585568125b1
718146 examples [00:48, 14431.60 examples/s]07/26/2021 00:32:32 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/f9b566f31181a53d426a2dc982a1b1de06cc92541de83cee688e5c57f4874300
1471415 examples [01:36, 13302.22 examples/s]07/26/2021 00:33:21 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/21f0672cc841442e067c7ea57471788dbd350f889acbd8028e75edb9efcacddb
2229278 examples [02:24, 16466.88 examples/s]07/26/2021 00:34:09 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/c027123c743fb1e0079bcd3be75f0ba6be89c6997f6b000e97c33f9c3d9c2742
2997743 examples [03:13, 18057.68 examples/s]07/26/2021 00:34:58 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/d7cc7a7389a8187b043cf359794e6fdc7783d5d0b6e7d737381e89d34c25e441
3772944 examples [04:02, 15671.97 examples/s]07/26/2021 00:35:46 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/a0175299b2eb4767f27e4f73c6848609be453fa5eb8d36dd6f8ecfd2c60a1e01
4569497 examples [04:51, 18017.92 examples/s]07/26/2021 00:36:35 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/6b432b7a552ccc65da0810808506bb7570162447776507b2b47319a230b48aa3
5356241 examples [05:39, 16205.13 examples/s]07/26/2021 00:37:24 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/ef34899af5cac3b75a798286fad2be831177c0833dab12c19c139b694d8c3544
6151458 examples [06:29, 11766.89 examples/s]07/26/2021 00:38:14 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/9926c88e0b8a2013f57aaef129cb9978ff129b8bfb3408c1194852c806249f9d
6957212 examples [07:18, 18684.33 examples/s]07/26/2021 00:39:03 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/aae79457ef2f44cd9ef24584b894c033d9099e6bc8e15b661a349cc185a230d7
7763558 examples [08:07, 16309.71 examples/s]07/26/2021 00:39:52 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/0274c31e96e2728161263b15aa4da982825eec91c7b0693756a890e76d1167c4
8565051 examples [08:57, 17289.47 examples/s]07/26/2021 00:40:41 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/f6423f5486261f771097352c7e2ae07643ad0f2fcf5f5d68c6a9921f8bd1e6a3
9397678 examples [09:46, 16643.61 examples/s]07/26/2021 00:41:30 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/2edc5ca535c1ea46aaacebf7f68a3553aa5d92b70e574f05709fa02dc52b5f4e
10231465 examples [10:36, 12871.41 examples/s]07/26/2021 00:42:20 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/3a06d248b02355ecdcf097df97a9e670db72c42456df9d04b15d4187933263ed
11075179 examples [11:26, 16567.73 examples/s]07/26/2021 00:43:11 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/0e3af1310ea118f4a5e8c13b40a561ae20ba209ae196d633a68155af35ec049c
Dataset oscar downloaded and prepared to /home/akali/.cache/huggingface/datasets/oscar/unshuffled_deduplicated_tr/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2. Subsequent calls will reuse this data.
07/26/2021 00:43:42 - WARNING - datasets.builder - Reusing dataset oscar (/home/akali/.cache/huggingface/datasets/oscar/unshuffled_deduplicated_tr/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2)
07/26/2021 00:43:42 - WARNING - run_mlm_no_trainer - You are instantiating a new config instance from scratch.
loading configuration file ./tokenizers/Roberta/config.json
Model config RobertaConfig {
  &quot;architectures&quot;: [
    &quot;RobertaForMaskedLM&quot;
  ],
  &quot;attention_probs_dropout_prob&quot;: 0.1,
  &quot;bos_token_id&quot;: 0,
  &quot;eos_token_id&quot;: 2,
  &quot;gradient_checkpointing&quot;: false,
  &quot;hidden_act&quot;: &quot;gelu&quot;,
  &quot;hidden_dropout_prob&quot;: 0.1,
  &quot;hidden_size&quot;: 768,
  &quot;initializer_range&quot;: 0.02,
  &quot;intermediate_size&quot;: 3072,
  &quot;layer_norm_eps&quot;: 1e-05,
  &quot;max_position_embeddings&quot;: 514,
  &quot;model_type&quot;: &quot;roberta&quot;,
  &quot;num_attention_heads&quot;: 12,
  &quot;num_hidden_layers&quot;: 12,
  &quot;pad_token_id&quot;: 1,
  &quot;position_embedding_type&quot;: &quot;absolute&quot;,
  &quot;transformers_version&quot;: &quot;4.10.0.dev0&quot;,
  &quot;type_vocab_size&quot;: 1,
  &quot;use_cache&quot;: true,
  &quot;vocab_size&quot;: 52000
}

Didn't find file ./tokenizers/Roberta/tokenizer.json. We won't load it.
Didn't find file ./tokenizers/Roberta/added_tokens.json. We won't load it.
loading file ./tokenizers/Roberta/vocab.json
loading file ./tokenizers/Roberta/merges.txt
loading file None
loading file None
loading file ./tokenizers/Roberta/special_tokens_map.json
loading file ./tokenizers/Roberta/tokenizer_config.json
loading configuration file ./tokenizers/Roberta/config.json
Model config RobertaConfig {
  &quot;architectures&quot;: [
    &quot;RobertaForMaskedLM&quot;
  ],
  &quot;attention_probs_dropout_prob&quot;: 0.1,
  &quot;bos_token_id&quot;: 0,
  &quot;eos_token_id&quot;: 2,
  &quot;gradient_checkpointing&quot;: false,
  &quot;hidden_act&quot;: &quot;gelu&quot;,
  &quot;hidden_dropout_prob&quot;: 0.1,
  &quot;hidden_size&quot;: 768,
  &quot;initializer_range&quot;: 0.02,
  &quot;intermediate_size&quot;: 3072,
  &quot;layer_norm_eps&quot;: 1e-05,
  &quot;max_position_embeddings&quot;: 514,
  &quot;model_type&quot;: &quot;roberta&quot;,
  &quot;num_attention_heads&quot;: 12,
  &quot;num_hidden_layers&quot;: 12,
  &quot;pad_token_id&quot;: 1,
  &quot;position_embedding_type&quot;: &quot;absolute&quot;,
  &quot;transformers_version&quot;: &quot;4.10.0.dev0&quot;,
  &quot;type_vocab_size&quot;: 1,
  &quot;use_cache&quot;: true,
  &quot;vocab_size&quot;: 52000
}

loading configuration file ./tokenizers/Roberta/config.json
Model config RobertaConfig {
  &quot;architectures&quot;: [
    &quot;RobertaForMaskedLM&quot;
  ],
  &quot;attention_probs_dropout_prob&quot;: 0.1,
  &quot;bos_token_id&quot;: 0,
  &quot;eos_token_id&quot;: 2,
  &quot;gradient_checkpointing&quot;: false,
  &quot;hidden_act&quot;: &quot;gelu&quot;,
  &quot;hidden_dropout_prob&quot;: 0.1,
  &quot;hidden_size&quot;: 768,
  &quot;initializer_range&quot;: 0.02,
  &quot;intermediate_size&quot;: 3072,
  &quot;layer_norm_eps&quot;: 1e-05,
  &quot;max_position_embeddings&quot;: 514,
  &quot;model_type&quot;: &quot;roberta&quot;,
  &quot;num_attention_heads&quot;: 12,
  &quot;num_hidden_layers&quot;: 12,
  &quot;pad_token_id&quot;: 1,
  &quot;position_embedding_type&quot;: &quot;absolute&quot;,
  &quot;transformers_version&quot;: &quot;4.10.0.dev0&quot;,
  &quot;type_vocab_size&quot;: 1,
  &quot;use_cache&quot;: true,
  &quot;vocab_size&quot;: 52000
}

# AFTER THIS POINT:
Script started to print tqdm process multiple times like that:
----&gt; LOOK HERE  Running tokenizer on dataset line_by_line #43:  19%|███████████████████████▏                                                                                               | 43/221 [12:20&lt;51:05, 17.22s/ba]
Running tokenizer on dataset line_by_line #36:  19%|███████████████████████▏                                                                                               | 43/221 [12:24&lt;51:20, 17.30s/ba]
Running tokenizer on dataset line_by_line #29:  19%|███████████████████████▏                                                                                               | 43/221 [12:28&lt;51:37, 17.40s/ba]
Running tokenizer on dataset line_by_line #38:  19%|███████████████████████▏                                                                                               | 43/221 [12:22&lt;51:15, 17.28s/ba]
Running tokenizer on dataset line_by_line #5:  18%|█████████████████████▏                                                                                                  | 39/221 [12:33&lt;58:34, 19.31s/ba]
Running tokenizer on dataset line_by_line #21:  19%|███████████████████████▏                                                                                               | 43/221 [12:30&lt;51:45, 17.45s/ba]
Running tokenizer on dataset line_by_line #46:  19%|███████████████████████▏                                                                                               | 43/221 [12:19&lt;51:01, 17.20s/ba]
Running tokenizer on dataset line_by_line #38:  19%|███████████████████████▏                                                                                               | 43/221 [12:25&lt;51:25, 17.34s/ba]
Running tokenizer on dataset line_by_line #42:  19%|███████████████████████▏                                                                                               | 43/221 [12:23&lt;51:19, 17.30s/ba]
Running tokenizer on dataset line_by_line #35:  19%|███████████████████████▏                                                                                               | 43/221 [12:26&lt;51:31, 17.37s/ba]
Running tokenizer on dataset line_by_line #21:  19%|███████████████████████▏                                                                                               | 43/221 [12:30&lt;51:48, 17.46s/ba]
Running tokenizer on dataset line_by_line #45:  19%|███████████████████████▏                                                                                               | 43/221 [12:23&lt;51:17, 17.29s/ba]
Running tokenizer on dataset line_by_line #35:  19%|███████████████████████▏                                                                                               | 43/221 [12:27&lt;51:34, 17.38s/ba]
----&gt; AND HERE Running tokenizer on dataset line_by_line #43:  18%|█████████████████████   

As it can be seen processor 43 printed 2 times but their percentage is inconsistent. Since it can't be decreased, I think it is preprocessing in for each core."><pre class="notranslate"><code class="notranslate">2021-07-26 00:30:54.355600: E tensorflow/core/framework/op_kernel.cc:1693] OpKernel ('op: "TPURoundRobin" device_type: "CPU"') for unknown op: TPURoundRobin
2021-07-26 00:30:54.355659: E tensorflow/core/framework/op_kernel.cc:1693] OpKernel ('op: "TpuHandleToProtoKey" device_type: "CPU"') for unknown op: TpuHandleToProtoKey
07/26/2021 00:31:13 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 0
Local process index: 0
Device: xla:1
Use FP16 precision: False

Downloading and preparing dataset oscar/unshuffled_deduplicated_tr (download: 9.68 GiB, generated: 26.43 GiB, post-processed: Unknown size, total: 36.10 GiB) to /home/akali/.cache/huggingface/datasets/oscar/unshuffled_deduplicated_tr/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
07/26/2021 00:31:20 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 1
Local process index: 1
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:20 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 5
Local process index: 5
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:20 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 7
Local process index: 7
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:20 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 6
Local process index: 6
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:21 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 2
Local process index: 2
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:21 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 4
Local process index: 4
Device: xla:0
Use FP16 precision: False

07/26/2021 00:31:23 - INFO - run_mlm_no_trainer - Distributed environment: TPU
Num processes: 8
Process index: 3
Local process index: 3
Device: xla:0
Use FP16 precision: False

0 examples [00:00, ? examples/s]07/26/2021 00:31:44 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/657d72dc352d822d0496bb9f519cf0de87b87064d56024d9d1ac5585568125b1
718146 examples [00:48, 14431.60 examples/s]07/26/2021 00:32:32 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/f9b566f31181a53d426a2dc982a1b1de06cc92541de83cee688e5c57f4874300
1471415 examples [01:36, 13302.22 examples/s]07/26/2021 00:33:21 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/21f0672cc841442e067c7ea57471788dbd350f889acbd8028e75edb9efcacddb
2229278 examples [02:24, 16466.88 examples/s]07/26/2021 00:34:09 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/c027123c743fb1e0079bcd3be75f0ba6be89c6997f6b000e97c33f9c3d9c2742
2997743 examples [03:13, 18057.68 examples/s]07/26/2021 00:34:58 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/d7cc7a7389a8187b043cf359794e6fdc7783d5d0b6e7d737381e89d34c25e441
3772944 examples [04:02, 15671.97 examples/s]07/26/2021 00:35:46 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/a0175299b2eb4767f27e4f73c6848609be453fa5eb8d36dd6f8ecfd2c60a1e01
4569497 examples [04:51, 18017.92 examples/s]07/26/2021 00:36:35 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/6b432b7a552ccc65da0810808506bb7570162447776507b2b47319a230b48aa3
5356241 examples [05:39, 16205.13 examples/s]07/26/2021 00:37:24 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/ef34899af5cac3b75a798286fad2be831177c0833dab12c19c139b694d8c3544
6151458 examples [06:29, 11766.89 examples/s]07/26/2021 00:38:14 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/9926c88e0b8a2013f57aaef129cb9978ff129b8bfb3408c1194852c806249f9d
6957212 examples [07:18, 18684.33 examples/s]07/26/2021 00:39:03 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/aae79457ef2f44cd9ef24584b894c033d9099e6bc8e15b661a349cc185a230d7
7763558 examples [08:07, 16309.71 examples/s]07/26/2021 00:39:52 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/0274c31e96e2728161263b15aa4da982825eec91c7b0693756a890e76d1167c4
8565051 examples [08:57, 17289.47 examples/s]07/26/2021 00:40:41 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/f6423f5486261f771097352c7e2ae07643ad0f2fcf5f5d68c6a9921f8bd1e6a3
9397678 examples [09:46, 16643.61 examples/s]07/26/2021 00:41:30 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/2edc5ca535c1ea46aaacebf7f68a3553aa5d92b70e574f05709fa02dc52b5f4e
10231465 examples [10:36, 12871.41 examples/s]07/26/2021 00:42:20 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/3a06d248b02355ecdcf097df97a9e670db72c42456df9d04b15d4187933263ed
11075179 examples [11:26, 16567.73 examples/s]07/26/2021 00:43:11 - INFO - datasets_modules.datasets.oscar.84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2.oscar - generating examples from = /home/akali/.cache/huggingface/datasets/downloads/0e3af1310ea118f4a5e8c13b40a561ae20ba209ae196d633a68155af35ec049c
Dataset oscar downloaded and prepared to /home/akali/.cache/huggingface/datasets/oscar/unshuffled_deduplicated_tr/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2. Subsequent calls will reuse this data.
07/26/2021 00:43:42 - WARNING - datasets.builder - Reusing dataset oscar (/home/akali/.cache/huggingface/datasets/oscar/unshuffled_deduplicated_tr/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2)
07/26/2021 00:43:42 - WARNING - run_mlm_no_trainer - You are instantiating a new config instance from scratch.
loading configuration file ./tokenizers/Roberta/config.json
Model config RobertaConfig {
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "transformers_version": "4.10.0.dev0",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 52000
}

Didn't find file ./tokenizers/Roberta/tokenizer.json. We won't load it.
Didn't find file ./tokenizers/Roberta/added_tokens.json. We won't load it.
loading file ./tokenizers/Roberta/vocab.json
loading file ./tokenizers/Roberta/merges.txt
loading file None
loading file None
loading file ./tokenizers/Roberta/special_tokens_map.json
loading file ./tokenizers/Roberta/tokenizer_config.json
loading configuration file ./tokenizers/Roberta/config.json
Model config RobertaConfig {
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "transformers_version": "4.10.0.dev0",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 52000
}

loading configuration file ./tokenizers/Roberta/config.json
Model config RobertaConfig {
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "transformers_version": "4.10.0.dev0",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 52000
}

# AFTER THIS POINT:
Script started to print tqdm process multiple times like that:
----&gt; LOOK HERE  Running tokenizer on dataset line_by_line #43:  19%|███████████████████████▏                                                                                               | 43/221 [12:20&lt;51:05, 17.22s/ba]
Running tokenizer on dataset line_by_line #36:  19%|███████████████████████▏                                                                                               | 43/221 [12:24&lt;51:20, 17.30s/ba]
Running tokenizer on dataset line_by_line #29:  19%|███████████████████████▏                                                                                               | 43/221 [12:28&lt;51:37, 17.40s/ba]
Running tokenizer on dataset line_by_line #38:  19%|███████████████████████▏                                                                                               | 43/221 [12:22&lt;51:15, 17.28s/ba]
Running tokenizer on dataset line_by_line #5:  18%|█████████████████████▏                                                                                                  | 39/221 [12:33&lt;58:34, 19.31s/ba]
Running tokenizer on dataset line_by_line #21:  19%|███████████████████████▏                                                                                               | 43/221 [12:30&lt;51:45, 17.45s/ba]
Running tokenizer on dataset line_by_line #46:  19%|███████████████████████▏                                                                                               | 43/221 [12:19&lt;51:01, 17.20s/ba]
Running tokenizer on dataset line_by_line #38:  19%|███████████████████████▏                                                                                               | 43/221 [12:25&lt;51:25, 17.34s/ba]
Running tokenizer on dataset line_by_line #42:  19%|███████████████████████▏                                                                                               | 43/221 [12:23&lt;51:19, 17.30s/ba]
Running tokenizer on dataset line_by_line #35:  19%|███████████████████████▏                                                                                               | 43/221 [12:26&lt;51:31, 17.37s/ba]
Running tokenizer on dataset line_by_line #21:  19%|███████████████████████▏                                                                                               | 43/221 [12:30&lt;51:48, 17.46s/ba]
Running tokenizer on dataset line_by_line #45:  19%|███████████████████████▏                                                                                               | 43/221 [12:23&lt;51:17, 17.29s/ba]
Running tokenizer on dataset line_by_line #35:  19%|███████████████████████▏                                                                                               | 43/221 [12:27&lt;51:34, 17.38s/ba]
----&gt; AND HERE Running tokenizer on dataset line_by_line #43:  18%|█████████████████████   

As it can be seen processor 43 printed 2 times but their percentage is inconsistent. Since it can't be decreased, I think it is preprocessing in for each core.
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">I expected to run training script with 8 cores with normal speed. But it is stoped at this point and not continue from here even without small changes.</p>
<p dir="auto">Hello,</p>
<p dir="auto">Suppose that I have two GPT2DoubleHeadsModel (let’s call it model A and B).<br>
Is there any way that I can:</p>
<ol dir="auto">
<li>take the hidden state of a given input at the n-th layer of the model A and feed it directly into the output layer of model B to compute output</li>
</ol>
<p dir="auto">AND</p>
<ol start="2" dir="auto">
<li>Take the output obtained from 1. and calculate the loss based on an appropriate label</li>
</ol>
<p dir="auto">Some coding example would be a great help.</p>
<p dir="auto">Thank you,</p>