<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.1.1</li>
<li>Platform: AWS Sagemaker</li>
<li>Python version:  3.6</li>
<li>PyTorch version (GPU?): 1.6</li>
<li>Tensorflow version (GPU?):</li>
<li>Using GPU in script?: YES</li>
<li>Using distributed or parallel set-up in script?: YES</li>
</ul>
<h3 dir="auto">Who can help</h3>

<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a>  <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/alexorona/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/alexorona">@alexorona</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): GPT2</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)</li>
</ul>
<p dir="auto">I am fine-tuning GPT2 using my own dataset, using the examples/language-modelling/run_clm.py script, and I want to use the new model_parallel feature in v4.1.1. I am using a multi-gpu instance (AWS p3.8xlarge - with 4 gpus).</p>
<p dir="auto">But I get this error:<br>
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [2], output_device 2, and module parameters {device(type='cpu')}.</p>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Run run_clm.py with the following params:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -m torch.distributed.launch \
    --nproc_per_node 4 run_clm.py \
    --do_train \
    --do_eval \
    --fp16 \
    --logging_first_step \
    --model_parallel \
    --evaluation_strategy epoch \
    --logging_steps 50 \
    --model_name_or_path gpt2 \
    --model_type gpt2 \
    --num_train_epochs 1 \
    --output_dir /opt/ml/model/ \
    --per_device_eval_batch_size 2 \
    --per_device_train_batch_size 2 \
    --save_steps 50 \
    --save_total_limit 1 \
    --train_file /opt/ml/input/data/data/train.txt \
    --validation_file /opt/ml/input/data/data/val.txt"><pre class="notranslate"><code class="notranslate">python -m torch.distributed.launch \
    --nproc_per_node 4 run_clm.py \
    --do_train \
    --do_eval \
    --fp16 \
    --logging_first_step \
    --model_parallel \
    --evaluation_strategy epoch \
    --logging_steps 50 \
    --model_name_or_path gpt2 \
    --model_type gpt2 \
    --num_train_epochs 1 \
    --output_dir /opt/ml/model/ \
    --per_device_eval_batch_size 2 \
    --per_device_train_batch_size 2 \
    --save_steps 50 \
    --save_total_limit 1 \
    --train_file /opt/ml/input/data/data/train.txt \
    --validation_file /opt/ml/input/data/data/val.txt
</code></pre></div>
<ol start="2" dir="auto">
<li>I get this error when training starts:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;run_clm.py&quot;, line 374, in &lt;module&gt;
    main()
  File &quot;run_clm.py&quot;, line 344, in main
    trainer.train(model_path=model_path)
  File &quot;/opt/conda/lib/python3.6/site-packages/transformers/trainer.py&quot;, line 681, in train
    else True
  File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 282, in __init__
#015 95%|â��â��â��â��â��â��â��â��â��â��| 62/65 [00:04&lt;00:00, 12.58ba/s]    ).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [1], output_device 1, and module parameters {device(type='cpu')}.
#015 98%|â��â��â��â��â��â��â��â��â��â��| 64/65 [00:05&lt;00:00, 12.26ba/s]#015100%|â��â��â��â��â��â��â��â��â��â��| 65/65 [00:05&lt;00:00, 12.61ba/s]
Traceback (most recent call last):
  File &quot;run_clm.py&quot;, line 374, in &lt;module&gt;
    main()
  File &quot;run_clm.py&quot;, line 344, in main
    trainer.train(model_path=model_path)
  File &quot;/opt/conda/lib/python3.6/site-packages/transformers/trainer.py&quot;, line 681, in train
    else True
  File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 282, in __init__
    ).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [2], output_device 2, and module parameters {device(type='cpu')}.
#015 95%|â��â��â��â��â��â��â��â��â��â��| 62/65 [00:05&lt;00:00, 11.84ba/s]#015 98%|â��â��â��â��â��â��â��â��â��â��| 64/65 [00:05&lt;00:00, 12.18ba/s]#015100%|â��â��â��â��â��â��â��â��â��â��| 65/65 [00:05&lt;00:00, 12.60ba/s]
Traceback (most recent call last):
  File &quot;run_clm.py&quot;, line 374, in &lt;module&gt;
    main()
  File &quot;run_clm.py&quot;, line 344, in main
    trainer.train(model_path=model_path)
  File &quot;/opt/conda/lib/python3.6/site-packages/transformers/trainer.py&quot;, line 681, in train
    else True
  File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 282, in __init__
    ).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [3], output_device 3, and module parameters {device(type='cpu')}.
#015 98%|â��â��â��â��â��â��â��â��â��â��| 64/65 [00:05&lt;00:00, 11.50ba/s]#015100%|â��â��â��â��â��â��â��â��â��â��| 65/65 [00:05&lt;00:00, 12.48ba/s]
[INFO|trainer.py:388] 2020-12-22 00:54:34,892 &gt;&gt; The following columns in the training set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
[INFO|trainer.py:388] 2020-12-22 00:54:34,892 &gt;&gt; The following columns in the evaluation set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
Traceback (most recent call last):
  File &quot;run_clm.py&quot;, line 374, in &lt;module&gt;
    main()
  File &quot;run_clm.py&quot;, line 344, in main
    trainer.train(model_path=model_path)
  File &quot;/opt/conda/lib/python3.6/site-packages/transformers/trainer.py&quot;, line 681, in train
    else True
  File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 282, in __init__
    ).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [0], output_device 0, and module parameters {device(type='cpu')}.
Traceback (most recent call last):
  File &quot;/opt/conda/lib/python3.6/runpy.py&quot;, line 193, in _run_module_as_main
    &quot;__main__&quot;, mod_spec)
  File &quot;/opt/conda/lib/python3.6/runpy.py&quot;, line 85, in _run_code
    exec(code, run_globals)
  File &quot;/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py&quot;, line 261, in &lt;module&gt;
    main()
  File &quot;/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py&quot;, line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'run_clm.py', '--local_rank=3', '--do_train', '--do_eval', '--fp16', '--logging_first_step', '--model_parallel', '--evaluation_strategy', 'epoch', '--logging_steps', '50', '--model_name_or_path', 'gpt2', '--model_type', 'gpt2', '--num_train_epochs', '1', '--output_dir', '/opt/ml/model/', '--per_device_eval_batch_size', '2', '--per_device_train_batch_size', '2', '--save_steps', '50', '--save_total_limit', '1', '--train_file', '/opt/ml/input/data/data/train.txt', '--validation_file', '/opt/ml/input/data/data/val.txt']' returned non-zero exit status 1."><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "run_clm.py", line 374, in &lt;module&gt;
    main()
  File "run_clm.py", line 344, in main
    trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
    else True
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
#015 95%|â��â��â��â��â��â��â��â��â��â��| 62/65 [00:04&lt;00:00, 12.58ba/s]    ).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [1], output_device 1, and module parameters {device(type='cpu')}.
#015 98%|â��â��â��â��â��â��â��â��â��â��| 64/65 [00:05&lt;00:00, 12.26ba/s]#015100%|â��â��â��â��â��â��â��â��â��â��| 65/65 [00:05&lt;00:00, 12.61ba/s]
Traceback (most recent call last):
  File "run_clm.py", line 374, in &lt;module&gt;
    main()
  File "run_clm.py", line 344, in main
    trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
    else True
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
    ).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [2], output_device 2, and module parameters {device(type='cpu')}.
#015 95%|â��â��â��â��â��â��â��â��â��â��| 62/65 [00:05&lt;00:00, 11.84ba/s]#015 98%|â��â��â��â��â��â��â��â��â��â��| 64/65 [00:05&lt;00:00, 12.18ba/s]#015100%|â��â��â��â��â��â��â��â��â��â��| 65/65 [00:05&lt;00:00, 12.60ba/s]
Traceback (most recent call last):
  File "run_clm.py", line 374, in &lt;module&gt;
    main()
  File "run_clm.py", line 344, in main
    trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
    else True
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
    ).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [3], output_device 3, and module parameters {device(type='cpu')}.
#015 98%|â��â��â��â��â��â��â��â��â��â��| 64/65 [00:05&lt;00:00, 11.50ba/s]#015100%|â��â��â��â��â��â��â��â��â��â��| 65/65 [00:05&lt;00:00, 12.48ba/s]
[INFO|trainer.py:388] 2020-12-22 00:54:34,892 &gt;&gt; The following columns in the training set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
[INFO|trainer.py:388] 2020-12-22 00:54:34,892 &gt;&gt; The following columns in the evaluation set don't have a corresponding argument in `GPT2LMHeadModel.forward` and have been ignored: .
Traceback (most recent call last):
  File "run_clm.py", line 374, in &lt;module&gt;
    main()
  File "run_clm.py", line 344, in main
    trainer.train(model_path=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 681, in train
    else True
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 282, in __init__
    ).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [0], output_device 0, and module parameters {device(type='cpu')}.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 261, in &lt;module&gt;
    main()
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/launch.py", line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'run_clm.py', '--local_rank=3', '--do_train', '--do_eval', '--fp16', '--logging_first_step', '--model_parallel', '--evaluation_strategy', 'epoch', '--logging_steps', '50', '--model_name_or_path', 'gpt2', '--model_type', 'gpt2', '--num_train_epochs', '1', '--output_dir', '/opt/ml/model/', '--per_device_eval_batch_size', '2', '--per_device_train_batch_size', '2', '--save_steps', '50', '--save_total_limit', '1', '--train_file', '/opt/ml/input/data/data/train.txt', '--validation_file', '/opt/ml/input/data/data/val.txt']' returned non-zero exit status 1.
</code></pre></div>

<h2 dir="auto">Expected behavior</h2>

<p dir="auto">I expect the fine-tuning script to run successfully.</p>
<p dir="auto">If I remove the --model_parallel in the args, then it does run successfully in distributed mode. But I want to use this new feature to reduce memory usage, and increase batch_size</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="question" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/2753.png">❓</g-emoji> Questions &amp; Help</h1>


<p dir="auto">I saw that "dynamic masking" was mentioned on the README file for language modeling:<br>
"In accordance to the RoBERTa paper, we use dynamic masking rather than static masking. The model may, therefore, converge slightly slower (over-fitting takes more epochs)."</p>
<p dir="auto">I couldn't find which class this method is implemented in and how to enable this feature during pre-training using the Trainer class. Could someone please help me?</p>
<p dir="auto">Thank you very much in advance.</p>