<h1 dir="auto">❓ Questions &amp; Help</h1>

<h2 dir="auto">Details</h2>

<p dir="auto">Is there a way to generate all possible sentences using a fine-tuned GPT-2 model given a certain sampling technique? For some reason I wan to exhaust all possible combination of tokens given a fine-tuned GPT-2 model with certain sampling technique. Is it doable? If it is not, how do we get an estimate of how many possible sentences are there in the latent space?</p>

<p dir="auto"><strong>A link to original question on Stack Overflow</strong>:</p>
<p dir="auto">I have a custom Bert model for relation classification based on the <a href="https://arxiv.org/abs/1905.08284" rel="nofollow">R-BERT paper</a>. The model performs well but is relatively slow on CPU, so I'd like to try exporting to ONNX.</p>
<p dir="auto">The model inherits from <code class="notranslate">BertPreTrainedModel</code> and is relatively simple:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class BertForRelationClassification(BertPreTrainedModel):
    def __init__(self, config):
        super(BertForRelationClassification, self).__init__(config)
        self.num_labels = config.num_labels
        self.bert = BertModel(config)
        self.cls_dropout = nn.Dropout(0.1)  
        self.ent_dropout = nn.Dropout(0.1)
        self.classifier = nn.Linear(config.hidden_size*3, self.config.num_labels)
        self.init_weights()

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, e1_mask=None, e2_mask=None, 
                        labels=None, position_ids=None, head_mask=None):
        outputs = self.bert(input_ids, position_ids=position_ids, token_type_ids=token_type_ids,
                            attention_mask=attention_mask, head_mask=head_mask)
        pooled_output = outputs[1]  
        sequence_output = outputs[0] 

        def extract_entity(sequence_output, e_mask):
            extended_e_mask = e_mask.unsqueeze(1)
            extended_e_mask = torch.bmm(
                extended_e_mask.float(), sequence_output).squeeze(1)
            return extended_e_mask.float()

        e1_h = self.ent_dropout(extract_entity(sequence_output, e1_mask))
        e2_h = self.ent_dropout(extract_entity(sequence_output, e2_mask))
        context = self.cls_dropout(pooled_output)
        pooled_output = torch.cat([context, e1_h, e2_h], dim=-1)

        logits = self.classifier(pooled_output)
        outputs = (logits,) + outputs[2:]

        if labels is not None:
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))

        outputs = (loss,) + outputs
        return outputs "><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">BertForRelationClassification</span>(<span class="pl-v">BertPreTrainedModel</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">config</span>):
        <span class="pl-en">super</span>(<span class="pl-v">BertForRelationClassification</span>, <span class="pl-s1">self</span>).<span class="pl-en">__init__</span>(<span class="pl-s1">config</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">num_labels</span> <span class="pl-c1">=</span> <span class="pl-s1">config</span>.<span class="pl-s1">num_labels</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">bert</span> <span class="pl-c1">=</span> <span class="pl-v">BertModel</span>(<span class="pl-s1">config</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">cls_dropout</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Dropout</span>(<span class="pl-c1">0.1</span>)  
        <span class="pl-s1">self</span>.<span class="pl-s1">ent_dropout</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Dropout</span>(<span class="pl-c1">0.1</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">classifier</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Linear</span>(<span class="pl-s1">config</span>.<span class="pl-s1">hidden_size</span><span class="pl-c1">*</span><span class="pl-c1">3</span>, <span class="pl-s1">self</span>.<span class="pl-s1">config</span>.<span class="pl-s1">num_labels</span>)
        <span class="pl-s1">self</span>.<span class="pl-en">init_weights</span>()

    <span class="pl-k">def</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input_ids</span>, <span class="pl-s1">token_type_ids</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">attention_mask</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">e1_mask</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">e2_mask</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, 
                        <span class="pl-s1">labels</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">position_ids</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">head_mask</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
        <span class="pl-s1">outputs</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">bert</span>(<span class="pl-s1">input_ids</span>, <span class="pl-s1">position_ids</span><span class="pl-c1">=</span><span class="pl-s1">position_ids</span>, <span class="pl-s1">token_type_ids</span><span class="pl-c1">=</span><span class="pl-s1">token_type_ids</span>,
                            <span class="pl-s1">attention_mask</span><span class="pl-c1">=</span><span class="pl-s1">attention_mask</span>, <span class="pl-s1">head_mask</span><span class="pl-c1">=</span><span class="pl-s1">head_mask</span>)
        <span class="pl-s1">pooled_output</span> <span class="pl-c1">=</span> <span class="pl-s1">outputs</span>[<span class="pl-c1">1</span>]  
        <span class="pl-s1">sequence_output</span> <span class="pl-c1">=</span> <span class="pl-s1">outputs</span>[<span class="pl-c1">0</span>] 

        <span class="pl-k">def</span> <span class="pl-en">extract_entity</span>(<span class="pl-s1">sequence_output</span>, <span class="pl-s1">e_mask</span>):
            <span class="pl-s1">extended_e_mask</span> <span class="pl-c1">=</span> <span class="pl-s1">e_mask</span>.<span class="pl-en">unsqueeze</span>(<span class="pl-c1">1</span>)
            <span class="pl-s1">extended_e_mask</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">bmm</span>(
                <span class="pl-s1">extended_e_mask</span>.<span class="pl-en">float</span>(), <span class="pl-s1">sequence_output</span>).<span class="pl-en">squeeze</span>(<span class="pl-c1">1</span>)
            <span class="pl-k">return</span> <span class="pl-s1">extended_e_mask</span>.<span class="pl-en">float</span>()

        <span class="pl-s1">e1_h</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">ent_dropout</span>(<span class="pl-en">extract_entity</span>(<span class="pl-s1">sequence_output</span>, <span class="pl-s1">e1_mask</span>))
        <span class="pl-s1">e2_h</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">ent_dropout</span>(<span class="pl-en">extract_entity</span>(<span class="pl-s1">sequence_output</span>, <span class="pl-s1">e2_mask</span>))
        <span class="pl-s1">context</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">cls_dropout</span>(<span class="pl-s1">pooled_output</span>)
        <span class="pl-s1">pooled_output</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">cat</span>([<span class="pl-s1">context</span>, <span class="pl-s1">e1_h</span>, <span class="pl-s1">e2_h</span>], <span class="pl-s1">dim</span><span class="pl-c1">=</span><span class="pl-c1">-</span><span class="pl-c1">1</span>)

        <span class="pl-s1">logits</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">classifier</span>(<span class="pl-s1">pooled_output</span>)
        <span class="pl-s1">outputs</span> <span class="pl-c1">=</span> (<span class="pl-s1">logits</span>,) <span class="pl-c1">+</span> <span class="pl-s1">outputs</span>[<span class="pl-c1">2</span>:]

        <span class="pl-k">if</span> <span class="pl-s1">labels</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>:
            <span class="pl-s1">loss_fct</span> <span class="pl-c1">=</span> <span class="pl-v">CrossEntropyLoss</span>()
            <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-en">loss_fct</span>(<span class="pl-s1">logits</span>.<span class="pl-en">view</span>(<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">num_labels</span>), <span class="pl-s1">labels</span>.<span class="pl-en">view</span>(<span class="pl-c1">-</span><span class="pl-c1">1</span>))

        <span class="pl-s1">outputs</span> <span class="pl-c1">=</span> (<span class="pl-s1">loss</span>,) <span class="pl-c1">+</span> <span class="pl-s1">outputs</span>
        <span class="pl-k">return</span> <span class="pl-s1">outputs</span> </pre></div>
<p dir="auto">Following the <a href="https://huggingface.co/transformers/serialization.html#implementing-a-custom-configuration-for-an-unsupported-architecture" rel="nofollow">guidelines for exporting custom models to ONNX</a>, I've created a custom OnnxConfig for it and specified inputs and outputs:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
class BertForRelationClassificationOnnxConfig(OnnxConfig):
    @property
    def inputs(self) -&gt; Mapping[str, Mapping[int, str]]:
        return OrderedDict(
            [
                (&quot;input_ids&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;}),
                (&quot;attention_mask&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;}),
                (&quot;token_type_ids&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;}),
                (&quot;labels&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;}),
                (&quot;e1_mask&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;}),
                (&quot;e2_mask&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;}),
            ]
        )

    @property
    def outputs(self) -&gt; Mapping[str, Mapping[int, str]]:
        return OrderedDict([(&quot;outputs&quot;, {0: &quot;batch&quot;, 1: &quot;sequence&quot;})]) "><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">BertForRelationClassificationOnnxConfig</span>(<span class="pl-v">OnnxConfig</span>):
    <span class="pl-en">@<span class="pl-s1">property</span></span>
    <span class="pl-k">def</span> <span class="pl-en">inputs</span>(<span class="pl-s1">self</span>) <span class="pl-c1">-&gt;</span> <span class="pl-v">Mapping</span>[<span class="pl-s1">str</span>, <span class="pl-v">Mapping</span>[<span class="pl-s1">int</span>, <span class="pl-s1">str</span>]]:
        <span class="pl-k">return</span> <span class="pl-v">OrderedDict</span>(
            [
                (<span class="pl-s">"input_ids"</span>, {<span class="pl-c1">0</span>: <span class="pl-s">"batch"</span>, <span class="pl-c1">1</span>: <span class="pl-s">"sequence"</span>}),
                (<span class="pl-s">"attention_mask"</span>, {<span class="pl-c1">0</span>: <span class="pl-s">"batch"</span>, <span class="pl-c1">1</span>: <span class="pl-s">"sequence"</span>}),
                (<span class="pl-s">"token_type_ids"</span>, {<span class="pl-c1">0</span>: <span class="pl-s">"batch"</span>, <span class="pl-c1">1</span>: <span class="pl-s">"sequence"</span>}),
                (<span class="pl-s">"labels"</span>, {<span class="pl-c1">0</span>: <span class="pl-s">"batch"</span>, <span class="pl-c1">1</span>: <span class="pl-s">"sequence"</span>}),
                (<span class="pl-s">"e1_mask"</span>, {<span class="pl-c1">0</span>: <span class="pl-s">"batch"</span>, <span class="pl-c1">1</span>: <span class="pl-s">"sequence"</span>}),
                (<span class="pl-s">"e2_mask"</span>, {<span class="pl-c1">0</span>: <span class="pl-s">"batch"</span>, <span class="pl-c1">1</span>: <span class="pl-s">"sequence"</span>}),
            ]
        )

    <span class="pl-en">@<span class="pl-s1">property</span></span>
    <span class="pl-k">def</span> <span class="pl-en">outputs</span>(<span class="pl-s1">self</span>) <span class="pl-c1">-&gt;</span> <span class="pl-v">Mapping</span>[<span class="pl-s1">str</span>, <span class="pl-v">Mapping</span>[<span class="pl-s1">int</span>, <span class="pl-s1">str</span>]]:
        <span class="pl-k">return</span> <span class="pl-v">OrderedDict</span>([(<span class="pl-s">"outputs"</span>, {<span class="pl-c1">0</span>: <span class="pl-s">"batch"</span>, <span class="pl-c1">1</span>: <span class="pl-s">"sequence"</span>})]) </pre></div>
<p dir="auto">However when I run <code class="notranslate">convert_graph_to_onnx.py</code>, (of course) the model is assumed to be <code class="notranslate">BertModel</code> and the inputs and outputs are those of the vanilla <code class="notranslate">BertModel</code>. I'm unclear on the next steps.</p>
<p dir="auto">I'm fairly sure this is what I should do next as stated on the documentation page: <em><strong>"Once this is done, a single step remains: adding this configuration object to the initialisation of the model class, and to the general <code class="notranslate">transformers</code> initialisation"</strong></em>.</p>
<p dir="auto">While I feel a bit dense I'm still not following how to make this work, as the <code class="notranslate">BertForRelationClassificationOnnxConfig</code> class I created doesn't inherit from <code class="notranslate">BertConfig</code> (I could make it do so, but the documentation doesn't specify this) so I don't see how I can use this for initialization of the model. The <a href="https://github.com/huggingface/transformers/pull/13049/commits/d097adcebd89a520f04352eb215a85916934204f">MBart example</a> doesn't make sense to me as I'm not contributing to the transformers code base.</p>
<p dir="auto">Can you please provide guidance or a specific example? Thank you!</p>