<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.3.2</li>
<li>Platform: linux</li>
<li>Python version: 3.7</li>
<li>PyTorch version (GPU?): 1.7</li>
<li>Tensorflow version (GPU?): -</li>
<li>Using GPU in script?: -</li>
<li>Using distributed or parallel set-up in script?: -</li>
</ul>
<h3 dir="auto">Who can help</h3>

<ul dir="auto">
<li>albert, bert, xlm: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a></li>
</ul>
<h2 dir="auto">Information</h2>
<p dir="auto">In this line </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/huggingface/transformers/blob/e73a3e1891775a915846cc0f24b7e9a26d6688fb/src/transformers/data/data_collator.py#L381">transformers/src/transformers/data/data_collator.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 381
      in
      <a data-pjax="true" class="commit-tease-sha" href="/huggingface/transformers/commit/e73a3e1891775a915846cc0f24b7e9a26d6688fb">e73a3e1</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L381" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="381"></td>
          <td id="LC381" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">indices_random</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">bernoulli</span>(<span class="pl-s1">torch</span>.<span class="pl-en">full</span>(<span class="pl-s1">labels</span>.<span class="pl-s1">shape</span>, <span class="pl-c1">0.5</span>)).<span class="pl-en">bool</span>() <span class="pl-c1">&amp;</span> <span class="pl-s1">masked_indices</span> <span class="pl-c1">&amp;</span> <span class="pl-c1">~</span><span class="pl-s1">indices_replaced</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
 you need to change 0.5 to 0.1 to match the description written that only in 10% you would like to change  tokens with randomly selected tokens.<p></p>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Nothing to reproduce.</p>

<h2 dir="auto">Expected behavior</h2>
<p dir="auto">probs should match BERT paper.</p>
<p dir="auto">I'm using an tweaked version of the <code class="notranslate">uer/roberta-base-chinese-extractive-qa</code> model. While I know how to train using multiple GPUs, it is not clear how to use multiple GPUs when coming to this stage. Essentially this is what I have:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline
os.environ[&quot;CUDA_VISIBLE_DEVICES&quot;] = &quot;0&quot;

model = AutoModelForQuestionAnswering.from_pretrained('uer/roberta-base-chinese-extractive-qa')
tokenizer = AutoTokenizer.from_pretrained('uer/roberta-base-chinese-extractive-qa')
QA = pipeline('question-answering', model=model, tokenizer=tokenizer, device=0)"><pre class="notranslate"><code class="notranslate">from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

model = AutoModelForQuestionAnswering.from_pretrained('uer/roberta-base-chinese-extractive-qa')
tokenizer = AutoTokenizer.from_pretrained('uer/roberta-base-chinese-extractive-qa')
QA = pipeline('question-answering', model=model, tokenizer=tokenizer, device=0)
</code></pre></div>
<p dir="auto">Except the model I'm using is a tweaked version of the base model shown above. I'm trying to process a large number of documents and at the rate it's going at it would take on the order of weeks. Is there anyway to set the pipeline to use multiple gpus? So I could use something like</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="os.environ[&quot;CUDA_VISIBLE_DEVICES&quot;] = &quot;0,1&quot;
device = torch.device(&quot;cuda&quot;)"><pre class="notranslate"><code class="notranslate">os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
device = torch.device("cuda")
</code></pre></div>
<p dir="auto">And then use both devices?</p>
<p dir="auto">Also looks like someone else had the same question: <a href="https://stackoverflow.com/questions/64193049/how-to-use-transformers-pipeline-with-multi-gpu" rel="nofollow">https://stackoverflow.com/questions/64193049/how-to-use-transformers-pipeline-with-multi-gpu</a></p>