<h2 dir="auto">❓ Questions &amp; Help</h2>
<p dir="auto">Hi,<br>
I am running a modified version of <code class="notranslate">run_lm_finetuning.py</code>, it was working fine and model checkpoints have been saved, until the last step of the first epoch (9677/9678), where I got this error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 9677/9678 [2:01:24&lt;00:00,  1.36it/s]
  File &quot;my_run_lm_finetuning.py&quot;, line 588, in &lt;module&gt;
    main()
  File &quot;my_run_lm_finetuning.py&quot;, line 542, in main
    global_step, tr_loss = train(args, train_dataset, model, bert_model_fintuned, tokenizer, bert_tokenizer)
  File &quot;my_run_lm_finetuning.py&quot;, line 260, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, enc_output, labels=labels)
  File &quot;/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py&quot;, line 489, in __call__
    result = self.forward(*input, **kwargs)
  File &quot;/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py&quot;, line 144, in forward
    return self.gather(outputs, self.output_device)
  File &quot;/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py&quot;, line 156, in gather
    return gather(outputs, output_device, dim=self.dim)
  File &quot;/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py&quot;, line 67, in gather
    return gather_map(outputs)
  File &quot;/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py&quot;, line 62, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File &quot;/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py&quot;, line 62, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File &quot;/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py&quot;, line 54, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File &quot;/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/_functions.py&quot;, line 68, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File &quot;/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/cuda/comm.py&quot;, line 166, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: Gather got an input of invalid size: got [2, 3, 12, 256, 64], but expected [2, 4, 12, 256, 64] (gather at /opt/conda/conda-bld/pytorch_1544199946412/work/torch/csrc/cuda/comm.cpp:227)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x45 (0x7f3c52b7fcc5 in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: torch::cuda::gather(c10::ArrayRef&lt;at::Tensor&gt;, long, c10::optional&lt;int&gt;) + 0x4d8 (0x7f3c936eaba8 in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #2: &lt;unknown function&gt; + 0x4f99de (0x7f3c936ed9de in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #3: &lt;unknown function&gt; + 0x111e36 (0x7f3c93305e36 in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;
frame #14: THPFunction_apply(_object*, _object*) + 0x5dd (0x7f3c9350140d in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 9677/9678 [2:01:24&lt;00:00,  1.36it/s]
  File "my_run_lm_finetuning.py", line 588, in &lt;module&gt;
    main()
  File "my_run_lm_finetuning.py", line 542, in main
    global_step, tr_loss = train(args, train_dataset, model, bert_model_fintuned, tokenizer, bert_tokenizer)
  File "my_run_lm_finetuning.py", line 260, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, enc_output, labels=labels)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 144, in forward
    return self.gather(outputs, self.output_device)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 156, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 67, in gather
    return gather_map(outputs)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 54, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 68, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File "/home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/cuda/comm.py", line 166, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: Gather got an input of invalid size: got [2, 3, 12, 256, 64], but expected [2, 4, 12, 256, 64] (gather at /opt/conda/conda-bld/pytorch_1544199946412/work/torch/csrc/cuda/comm.cpp:227)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x45 (0x7f3c52b7fcc5 in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: torch::cuda::gather(c10::ArrayRef&lt;at::Tensor&gt;, long, c10::optional&lt;int&gt;) + 0x4d8 (0x7f3c936eaba8 in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #2: &lt;unknown function&gt; + 0x4f99de (0x7f3c936ed9de in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #3: &lt;unknown function&gt; + 0x111e36 (0x7f3c93305e36 in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;
frame #14: THPFunction_apply(_object*, _object*) + 0x5dd (0x7f3c9350140d in /home/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
</code></pre></div>
<p dir="auto">Note that in this experiment I used a fine-tuned version of Bert (I fine-tuned it using your previous script in lm_finetune folder) and there I have the <code class="notranslate">max_seq_length =256</code>, however when running this (<code class="notranslate">run_lm_finetuning.py</code>) , I have <code class="notranslate">block_size=128</code>.</p>
<p dir="auto">Any idea of what is the error for?</p>
<h3 dir="auto">Feature request</h3>
<p dir="auto">Dear Hugging Face Team,</p>
<p dir="auto">My name is Yongbin Li. I am part of <a href="https://github.com/hpcaitech/ColossalAI">Colossal-AI</a> Team.</p>
<p dir="auto">Thanks for your previous <a href="https://github.com/hpcaitech/ColossalAI/issues/396" data-hovercard-type="issue" data-hovercard-url="/hpcaitech/ColossalAI/issues/396/hovercard">invitation</a> to Colossal-AI org to join Hugging Face. We are happy to share our founder's <a href="https://twitter.com/HPCAITech/status/1547041583337394176" rel="nofollow">blog </a>about Hugging Face.</p>
<p dir="auto">We are thinking about further collaboration, eg. integrating Colossal-AI into Hugging Face to help your community members use large AI models in an efficient and easier manner.</p>
<p dir="auto">For example, we can democratize its access to all your users in the same way as you did with DeepSpeed.<br>
<a href="https://huggingface.co/docs/transformers/v4.21.0/en/main_classes/deepspeed" rel="nofollow">https://huggingface.co/docs/transformers/v4.21.0/en/main_classes/deepspeed</a></p>
<h3 dir="auto">Motivation</h3>
<p dir="auto">We believe the democratization of large AI models is also very helpful for Hugging Face members. We are very appreciated if we could build the integration with you to benefit both of our users.</p>
<p dir="auto">Actually, we are working on similar integrations with Meta OPT(<a href="https://github.com/facebookresearch/metaseq#using-opt-with-colossal-ai">done</a>), PyTorch Lightning(<a href="https://github.com/hpcaitech/ColossalAI/issues/1330" data-hovercard-type="issue" data-hovercard-url="/hpcaitech/ColossalAI/issues/1330/hovercard">in process</a>), etc.</p>
<h3 dir="auto">Your contribution</h3>
<p dir="auto">We can provide help you need in this cooperation for free. Actually, we have reached a preliminary idea with your team member: omar, lysandre, and julien via email(<a href="mailto:ybl@hpcaitech.com">ybl@hpcaitech.com</a>) and look forward to your further reply.</p>
<p dir="auto">Feel free to reach out to me on Hugging Face Discord. My username is billy2022. We can discuss more details with other colleagues in a private group.</p>
<p dir="auto">Thank you very much.</p>
<p dir="auto">Best regards,<br>
Yongbin Li, Chief Marketing Officer, HPC-AI Tech</p>