<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/41872440/216619913-99568f8c-8a82-42ab-ae7a-dd56846dd395.png"><img src="https://user-images.githubusercontent.com/41872440/216619913-99568f8c-8a82-42ab-ae7a-dd56846dd395.png" alt="image" style="max-width: 100%;"></a><br>
I have 8 gpu's in this machine.<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/41872440/216619996-7af22496-7fb8-4b6d-90f2-3606846b3cdc.png"><img src="https://user-images.githubusercontent.com/41872440/216619996-7af22496-7fb8-4b6d-90f2-3606846b3cdc.png" alt="image" style="max-width: 100%;"></a><br>
I think its not taking all 8 gpu's. Already tried changing batch_sizes and with multiple of 8</p>
<p dir="auto"><em>Originally posted by @namanpundir in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1566659240" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/21407" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/21407/hovercard?comment_id=1415898415&amp;comment_type=issue_comment" href="https://github.com/huggingface/transformers/issues/21407#issuecomment-1415898415">#21407 (comment)</a></em></p>
<h3 dir="auto">System Info</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="- OS Platform and Distribution: Amazon Linux 2
- ONNX version 1.11.0 
- Python version: 3.8
- Transformers 4.17"><pre class="notranslate">- OS Platform and Distribution: Amazon Linux 2
- ONNX version 1.11.0 
- Python version: 3.8
- Transformers 4.17</pre></div>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patil-suraj/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patil-suraj">@patil-suraj</a></p>
<h3 dir="auto">Describe the bug</h3>
<p dir="auto">When I try and convert <code class="notranslate">microsoft/deberta-v3-large</code> to onnx format, the file size doubles to 1.8Gb while the pytorch file size is 800Mb.<br>
There was an issue <a href="https://github.com/onnx/onnx/issues/3278#issuecomment-781948998" data-hovercard-type="issue" data-hovercard-url="/onnx/onnx/issues/3278/hovercard">here</a> that was resolved by removing the duplication of weights. However, trying this code snippet in the deberta case only reduced the file size from 1.8Gb to 1.6Gb.</p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from collections import OrderedDict
from typing import Mapping
from pathlib import Path
from transformers.onnx import export
from transformers.onnx import OnnxConfig
from transformers import AutoTokenizer, AutoModel, AutoConfig

onnx_path = Path(&quot;deberta.onnx&quot;)

class DebertaConfig(OnnxConfig):
    @property
    def inputs(self) -&gt; Mapping[str, Mapping[int, str]]:
        return OrderedDict(
            [
                (&quot;input_ids&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;}),
                (&quot;attention_mask&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;}),
                (&quot;token_type_ids&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;}),
            ]
        )

config = AutoConfig.from_pretrained(&quot;microsoft/deberta-v3-large&quot;)

base_model = AutoModel.from_pretrained(&quot;microsoft/deberta-v3-large&quot;)
tokenizer = AutoTokenizer.from_pretrained(&quot;microsoft/deberta-v3-large&quot;)
onnx_config = DebertaConfig(config)
onnx_inputs, onnx_outputs = export(tokenizer, base_model, onnx_config, 15, onnx_path)"><pre class="notranslate"><code class="notranslate">from collections import OrderedDict
from typing import Mapping
from pathlib import Path
from transformers.onnx import export
from transformers.onnx import OnnxConfig
from transformers import AutoTokenizer, AutoModel, AutoConfig

onnx_path = Path("deberta.onnx")

class DebertaConfig(OnnxConfig):
    @property
    def inputs(self) -&gt; Mapping[str, Mapping[int, str]]:
        return OrderedDict(
            [
                ("input_ids", {0: "batch", 1: "sequence"}),
                ("attention_mask", {0: "batch", 1: "sequence"}),
                ("token_type_ids", {0: "batch", 1: "sequence"}),
            ]
        )

config = AutoConfig.from_pretrained("microsoft/deberta-v3-large")

base_model = AutoModel.from_pretrained("microsoft/deberta-v3-large")
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-large")
onnx_config = DebertaConfig(config)
onnx_inputs, onnx_outputs = export(tokenizer, base_model, onnx_config, 15, onnx_path)
</code></pre></div>
<h3 dir="auto">Expected behavior</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Expect the model.onnx file to be ~800Mb like the pytorch file."><pre class="notranslate">Expect the model.onnx file to be <span class="pl-k">~</span>800Mb like the pytorch file.</pre></div>