<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.15.0</li>
<li>Platform: Windows-10-10.0.19041-SP0</li>
<li>Python version: 3.8.5</li>
<li>PyTorch version (GPU?): 1.11.0+cu113 (True)</li>
<li>Tensorflow version (GPU?): 2.5.1 (True)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: yes</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Narsil/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Narsil">@Narsil</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import GPTNeoForCausalLM, GPT2Tokenizer

model_name = &quot;EleutherAI/gpt-neo-125M&quot;

model = GPTNeoForCausalLM.from_pretrained(model_name, low_cpu_mem_usage=True, cache_dir='gpt_cache_dir', resume_download=True).half().to(&quot;cuda:0&quot;)
tokenizer = GPT2Tokenizer.from_pretrained(model_name, low_cpu_mem_usage=True, cache_dir='gpt_cache_dir', resume_download=True)

input_ids = tokenizer(&quot;This is a line 1\n\nThis is a line 2\n\nThis is a line 3\n\n&quot;, return_tensors=&quot;pt&quot;).input_ids.cuda()
gen_tokens = model.generate(input_ids, do_sample=True, temperature=0.01, max_length=40, min_length=1, repetition_penalty=1.0)

gen_text = &quot;Output: \&quot;&quot; + tokenizer.batch_decode(gen_tokens[:, input_ids.shape[1]:])[0] + &quot;\&quot;&quot;

print(gen_text)"><pre class="notranslate"><code class="notranslate">from transformers import GPTNeoForCausalLM, GPT2Tokenizer

model_name = "EleutherAI/gpt-neo-125M"

model = GPTNeoForCausalLM.from_pretrained(model_name, low_cpu_mem_usage=True, cache_dir='gpt_cache_dir', resume_download=True).half().to("cuda:0")
tokenizer = GPT2Tokenizer.from_pretrained(model_name, low_cpu_mem_usage=True, cache_dir='gpt_cache_dir', resume_download=True)

input_ids = tokenizer("This is a line 1\n\nThis is a line 2\n\nThis is a line 3\n\n", return_tensors="pt").input_ids.cuda()
gen_tokens = model.generate(input_ids, do_sample=True, temperature=0.01, max_length=40, min_length=1, repetition_penalty=1.0)

gen_text = "Output: \"" + tokenizer.batch_decode(gen_tokens[:, input_ids.shape[1]:])[0] + "\""

print(gen_text)
</code></pre></div>
<p dir="auto">Actual behavior:<br>
-If the input ends with 1 newline, generating multiple tokens works as expected, but generating just 1 token says the next token should be a newline by itself.<br>
-If the input ends with 2 newlines, generate multiple tokens doesn't work as expected, and printing the next top score reveals the next token is some unexpected thing such as another newline or a token beginning with a space.</p>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">Expected behavior: If prompt ends in \n\n, generated text shouldn't start with \n.</p>
<p dir="auto">Duplicate of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1283293287" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/17860" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/17860/hovercard" href="https://github.com/huggingface/transformers/issues/17860">#17860</a> but it won't let me re-open</p>
<p dir="auto">Hi<br>
I am using transformer 4.1.1 on multiple gpus with distributed launch script below, the n_gpu set in this case is 1 instead of actual n_gpus, could you tell me how I can access to the total number of ranks calling the launcher with? Is there any variable in huggingface library which could show the number of gpus the script is called with? Here is the command I run:</p>
<p dir="auto"><code class="notranslate">export BS=4; CUDA_VISIBLE_DEVICES=0,1  USE_TF=0   python -m torch.distributed.launch --nproc_per_node=2 --master_port=9915  finetune_trainer.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler  --task translation --val_max_target_length 128 --warmup_steps 500 --n_train 500 </code></p>
<p dir="auto">thanks</p>