<p dir="auto">Hi,</p>
<p dir="auto">the albert tokenizer implements the <code class="notranslate">convert_tokens_to_string</code> function:</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/albert/tokenization_albert.py#L222-L223">transformers/src/transformers/models/albert/tokenization_albert.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 222 to 223
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182">ba0d50f</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L222" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="222"></td>
          <td id="LC222" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">out_string</span> <span class="pl-c1">=</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-s1">tokens</span>).<span class="pl-en">replace</span>(<span class="pl-v">SPIECE_UNDERLINE</span>, <span class="pl-s">" "</span>).<span class="pl-en">strip</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L223" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="223"></td>
          <td id="LC223" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">out_string</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">While the deberta v2 and some other tokenizer just delegate this to the sentencepiece tokenizer:</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/deberta_v2/tokenization_deberta_v2.py#L146">transformers/src/transformers/models/deberta_v2/tokenization_deberta_v2.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 146
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182">ba0d50f</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L146" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="146"></td>
          <td id="LC146" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">self</span>.<span class="pl-s1">_tokenizer</span>.<span class="pl-en">decode</span>(<span class="pl-s1">tokens</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">IMO it would be better to always delegate to the sentencepiece tokenizer. What do you think?</p>
<h2 dir="auto">PS:</h2>
<p dir="auto">Some more examples here</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/barthez/tokenization_barthez.py#L251-L252">transformers/src/transformers/models/barthez/tokenization_barthez.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 251 to 252
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182">ba0d50f</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L251" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="251"></td>
          <td id="LC251" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">out_string</span> <span class="pl-c1">=</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-s1">tokens</span>).<span class="pl-en">replace</span>(<span class="pl-v">SPIECE_UNDERLINE</span>, <span class="pl-s">" "</span>).<span class="pl-en">strip</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L252" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="252"></td>
          <td id="LC252" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">out_string</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/camembert/tokenization_camembert.py#L251-L252">transformers/src/transformers/models/camembert/tokenization_camembert.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 251 to 252
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182">ba0d50f</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L251" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="251"></td>
          <td id="LC251" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">out_string</span> <span class="pl-c1">=</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-s1">tokens</span>).<span class="pl-en">replace</span>(<span class="pl-v">SPIECE_UNDERLINE</span>, <span class="pl-s">" "</span>).<span class="pl-en">strip</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L252" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="252"></td>
          <td id="LC252" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">out_string</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/m2m_100/tokenization_m2m_100.py#L187-L188">transformers/src/transformers/models/m2m_100/tokenization_m2m_100.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 187 to 188
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182">ba0d50f</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L187" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="187"></td>
          <td id="LC187" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">out_string</span> <span class="pl-c1">=</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-s1">tokens</span>).<span class="pl-en">replace</span>(<span class="pl-v">SPIECE_UNDERLINE</span>, <span class="pl-s">" "</span>).<span class="pl-en">strip</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L188" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="188"></td>
          <td id="LC188" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">out_string</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/mbart/tokenization_mbart50.py#L208-L209">transformers/src/transformers/models/mbart/tokenization_mbart50.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 208 to 209
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182">ba0d50f</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L208" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="208"></td>
          <td id="LC208" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">out_string</span> <span class="pl-c1">=</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-s1">tokens</span>).<span class="pl-en">replace</span>(<span class="pl-v">SPIECE_UNDERLINE</span>, <span class="pl-s">" "</span>).<span class="pl-en">strip</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L209" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="209"></td>
          <td id="LC209" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">out_string</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/speech_to_text/tokenization_speech_to_text.py#L169-L173">transformers/src/transformers/models/speech_to_text/tokenization_speech_to_text.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 169 to 173
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182">ba0d50f</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L169" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="169"></td>
          <td id="LC169" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">out_string</span> <span class="pl-c1">=</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-s1">tokens</span>).<span class="pl-en">replace</span>(<span class="pl-v">SPIECE_UNDERLINE</span>, <span class="pl-s">" "</span>).<span class="pl-en">strip</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L170" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="170"></td>
          <td id="LC170" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">  </td>
        </tr>

        <tr class="border-0">
          <td id="L171" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="171"></td>
          <td id="LC171" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">do_upper_case</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L172" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="172"></td>
          <td id="LC172" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">out_string</span> <span class="pl-c1">=</span> <span class="pl-s1">out_string</span>.<span class="pl-en">upper</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L173" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="173"></td>
          <td id="LC173" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">out_string</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py#L264-L265">transformers/src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 264 to 265
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182">ba0d50f</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L264" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="264"></td>
          <td id="LC264" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">out_string</span> <span class="pl-c1">=</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-s1">tokens</span>).<span class="pl-en">replace</span>(<span class="pl-v">SPIECE_UNDERLINE</span>, <span class="pl-s">" "</span>).<span class="pl-en">strip</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L265" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="265"></td>
          <td id="LC265" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">return</span> <span class="pl-s1">out_string</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<h3 dir="auto">System Info</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="When I finetuned the text classification model based on the glue no trainer script, I found a bug in our script.
The URL is below:
https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue_no_trainer.py#L525

When we use the accelerator for multi-GPU training, the code should transfer from 
if step == len(eval_dataloader)
to 
if step == len(eval_dataloader) -1
Otherwise, it cannot work to filter the last step duplicated samples."><pre class="notranslate">When I finetuned the text classification model based on the glue no trainer script, I found a bug <span class="pl-k">in</span> our script.
The URL is below:
https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue_no_trainer.py#L525

When we use the accelerator <span class="pl-k">for</span> multi-GPU training, the code should transfer from 
<span class="pl-k">if</span> step == len(eval_dataloader)
to 
<span class="pl-k">if</span> step == len(eval_dataloader) -1
Otherwise, it cannot work to filter the last step duplicated samples.</pre></div>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">just run the script with a text classification using multi-GPU accelerator. The problem occurs in the last step for duplicated samples.</p>
<h3 dir="auto">Expected behavior</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="I think it should be fixed soon."><pre class="notranslate">I think it should be fixed soon.</pre></div>