<p dir="auto">It seems like <code class="notranslate">google/roberta2roberta_L-24_wikisplit</code> should be pre-processed differently than originally thought:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained(&quot;google/roberta2roberta_L-24_wikisplit&quot;)
model = AutoModelForSeq2SeqLM.from_pretrained(&quot;google/roberta2roberta_L-24_wikisplit&quot;)

long_sentence = &quot;&quot;&quot;Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it and he decides to open Bob 's Burgers for customers who were planning on going to Lobsterfest.&quot;&quot;&quot;

input_ids = tokenizer(long_sentence, return_tensors=&quot;pt&quot;).input_ids
output_ids = model.generate(input_ids)[0]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
# should output
# Due Due hurricane, Lobsterfest has been canceled, making Bob very happy about it. He decides to open B
# ob's Burgers for customers who were planning on going to Lobsterfest.com."><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoTokenizer</span>, <span class="pl-v">AutoModelForSeq2SeqLM</span>

<span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"google/roberta2roberta_L-24_wikisplit"</span>)
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">AutoModelForSeq2SeqLM</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"google/roberta2roberta_L-24_wikisplit"</span>)

<span class="pl-s1">long_sentence</span> <span class="pl-c1">=</span> <span class="pl-s">"""Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it and he decides to open Bob 's Burgers for customers who were planning on going to Lobsterfest."""</span>

<span class="pl-s1">input_ids</span> <span class="pl-c1">=</span> <span class="pl-en">tokenizer</span>(<span class="pl-s1">long_sentence</span>, <span class="pl-s1">return_tensors</span><span class="pl-c1">=</span><span class="pl-s">"pt"</span>).<span class="pl-s1">input_ids</span>
<span class="pl-s1">output_ids</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-en">generate</span>(<span class="pl-s1">input_ids</span>)[<span class="pl-c1">0</span>]
<span class="pl-en">print</span>(<span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>(<span class="pl-s1">output_ids</span>, <span class="pl-s1">skip_special_tokens</span><span class="pl-c1">=</span><span class="pl-c1">True</span>))
<span class="pl-c"># should output</span>
<span class="pl-c"># Due Due hurricane, Lobsterfest has been canceled, making Bob very happy about it. He decides to open B</span>
<span class="pl-c"># ob's Burgers for customers who were planning on going to Lobsterfest.com.</span></pre></div>
<p dir="auto">yields a weird word duplication bug, whereas:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained(&quot;google/roberta2roberta_L-24_wikisplit&quot;)
model = AutoModelForSeq2SeqLM.from_pretrained(&quot;google/roberta2roberta_L-24_wikisplit&quot;)

long_sentence = &quot;&quot;&quot;Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it and he decides to open Bob 's Burgers for customers who were planning on going to Lobsterfest.&quot;&quot;&quot;

input_ids = tokenizer(tokenizer.bos_token + long_sentence + tokenizer.eos_token, return_tensors=&quot;pt&quot;).input_ids
output_ids = model.generate(input_ids)[0]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
# should output
# Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it. He decides to open Bob's Burgers for customers who were planning on going to Lobsterfest.  "><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoTokenizer</span>, <span class="pl-v">AutoModelForSeq2SeqLM</span>

<span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-v">AutoTokenizer</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"google/roberta2roberta_L-24_wikisplit"</span>)
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">AutoModelForSeq2SeqLM</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">"google/roberta2roberta_L-24_wikisplit"</span>)

<span class="pl-s1">long_sentence</span> <span class="pl-c1">=</span> <span class="pl-s">"""Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it and he decides to open Bob 's Burgers for customers who were planning on going to Lobsterfest."""</span>

<span class="pl-s1">input_ids</span> <span class="pl-c1">=</span> <span class="pl-en">tokenizer</span>(<span class="pl-s1">tokenizer</span>.<span class="pl-s1">bos_token</span> <span class="pl-c1">+</span> <span class="pl-s1">long_sentence</span> <span class="pl-c1">+</span> <span class="pl-s1">tokenizer</span>.<span class="pl-s1">eos_token</span>, <span class="pl-s1">return_tensors</span><span class="pl-c1">=</span><span class="pl-s">"pt"</span>).<span class="pl-s1">input_ids</span>
<span class="pl-s1">output_ids</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-en">generate</span>(<span class="pl-s1">input_ids</span>)[<span class="pl-c1">0</span>]
<span class="pl-en">print</span>(<span class="pl-s1">tokenizer</span>.<span class="pl-en">decode</span>(<span class="pl-s1">output_ids</span>, <span class="pl-s1">skip_special_tokens</span><span class="pl-c1">=</span><span class="pl-c1">True</span>))
<span class="pl-c"># should output</span>
<span class="pl-c"># Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it. He decides to open Bob's Burgers for customers who were planning on going to Lobsterfest.  </span></pre></div>
<p dir="auto">yields good results.<br>
Notice the difference of<br>
<code class="notranslate">input_ids = tokenizer(tokenizer.bos_token + long_sentence + tokenizer.eos_token, return_tensors="pt").input_ids</code> between code sample 1 and 2.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a></p>
<p dir="auto">Wait for author's (<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/shashiongithub/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/shashiongithub">@shashiongithub</a>) answer before changing code example.</p>
<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.28.0.dev0</li>
<li>Platform: Linux-4.15.0-175-generic-x86_64-with-glibc2.27</li>
<li>Python version: 3.10.10</li>
<li>Huggingface_hub version: 0.13.4</li>
<li>Safetensors version: not installed</li>
<li>PyTorch version (GPU?): 1.12.0 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: </li>
<li>Using distributed or parallel set-up in script?: </li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/stas00/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/stas00">@stas00</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">I am training GPT on 2 nodes, each with 8 GPUs currently. I used the <code class="notranslate">Trainer</code> API provided by huggingface for training. In addition, I used Deepspeed's <code class="notranslate">ZeRO3</code> strategy. I have successfully start training, but there are the following issues in the output log:</p>
<ol dir="auto">
<li>Due to the presence of two nodes, there will be two rows of output at a time, each from a different node. I observed that the output of two nodes is the same, and the following is an example. Do I only need to focus on one line of output?(in other words, one of these two lines is redundant.) Or do these two lines mean that two machines have fed the same data and print the same output?
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="node1: {'loss': 1.4406, 'learning_rate': 2e-05, 'epoch': 0.17}
node2: {'loss': 1.4406, 'learning_rate': 2e-05, 'epoch': 0.17}
node1: {'loss': 1.4457, 'learning_rate': 2e-05, 'epoch': 0.18}
node2: {'loss': 1.4457, 'learning_rate': 2e-05, 'epoch': 0.18}"><pre class="notranslate">node1: {<span class="pl-s"><span class="pl-pds">'</span>loss<span class="pl-pds">'</span></span>: 1.4406, <span class="pl-s"><span class="pl-pds">'</span>learning_rate<span class="pl-pds">'</span></span>: 2e-05, <span class="pl-s"><span class="pl-pds">'</span>epoch<span class="pl-pds">'</span></span>: 0.17}
node2: {<span class="pl-s"><span class="pl-pds">'</span>loss<span class="pl-pds">'</span></span>: 1.4406, <span class="pl-s"><span class="pl-pds">'</span>learning_rate<span class="pl-pds">'</span></span>: 2e-05, <span class="pl-s"><span class="pl-pds">'</span>epoch<span class="pl-pds">'</span></span>: 0.17}
node1: {<span class="pl-s"><span class="pl-pds">'</span>loss<span class="pl-pds">'</span></span>: 1.4457, <span class="pl-s"><span class="pl-pds">'</span>learning_rate<span class="pl-pds">'</span></span>: 2e-05, <span class="pl-s"><span class="pl-pds">'</span>epoch<span class="pl-pds">'</span></span>: 0.18}
node2: {<span class="pl-s"><span class="pl-pds">'</span>loss<span class="pl-pds">'</span></span>: 1.4457, <span class="pl-s"><span class="pl-pds">'</span>learning_rate<span class="pl-pds">'</span></span>: 2e-05, <span class="pl-s"><span class="pl-pds">'</span>epoch<span class="pl-pds">'</span></span>: 0.18}</pre></div>
</li>
<li>There will be a progress bar during single node training, which should be performed using <code class="notranslate">tqdm</code>. However, there is no progress bar during the training.</li>
<li>In the training command, I used <code class="notranslate">--report_to "tensorboard"</code>, but I did not find any output in <code class="notranslate">tensorboard</code>.</li>
</ol>
<p dir="auto">Here is my command to start training.</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="deepspeed --num_gpus 8 --num_nodes 2  --hostfile=host.txt train.py \
    --model_name_or_path /path/to/model \
    --data_path /path/to/data \
    --output_dir /path/to/output \
    --num_train_epochs 1 \
    --per_device_train_batch_size 32 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy &quot;no&quot; \
    --save_strategy &quot;steps&quot; \
    --save_steps 100 \
    --save_total_limit 2 \
    --learning_rate 2e-5 \
    --logging_steps 1 \
    --report_to &quot;tensorboard&quot; \
    --gradient_checkpointing True \
    --deepspeed configs/deepspeed_config.json \
    --fp16 True"><pre class="notranslate">deepspeed --num_gpus 8 --num_nodes 2  --hostfile=host.txt train.py \
    --model_name_or_path /path/to/model \
    --data_path /path/to/data \
    --output_dir /path/to/output \
    --num_train_epochs 1 \
    --per_device_train_batch_size 32 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy <span class="pl-s"><span class="pl-pds">"</span>no<span class="pl-pds">"</span></span> \
    --save_strategy <span class="pl-s"><span class="pl-pds">"</span>steps<span class="pl-pds">"</span></span> \
    --save_steps 100 \
    --save_total_limit 2 \
    --learning_rate 2e-5 \
    --logging_steps 1 \
    --report_to <span class="pl-s"><span class="pl-pds">"</span>tensorboard<span class="pl-pds">"</span></span> \
    --gradient_checkpointing True \
    --deepspeed configs/deepspeed_config.json \
    --fp16 True</pre></div>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">I hope you can provide answers to the above three questions. Thanks!</p>