<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature request</h1>

<p dir="auto"><strong><em>"Disjunctive Positive Constraint Decoding"</em></strong> Algorithm proposed by a recent paper: <a href="https://www.ijcai.org/proceedings/2020/0502.pdf" rel="nofollow"><em>Guided Generation of Cause and Effect</em></a></p>
<p dir="auto">Currently the model.generation() method limits the user to just the highest probability outputs. Here the user is able to force diverse outputs by forcing the model to include diverse tokens across multiple generations.</p>
<p dir="auto">This method is called "Disjunctive Positive Constraint Decoding", and it forces the <code class="notranslate">model.generate()</code> process to generate sequences with the highest probabilities under the constraint of needing to include a set of provided tokens.</p>
<p dir="auto">This <strong>"disjunctive"</strong> method is powerful in that it can handle <strong>lemmatizing these forced tokens.</strong> For instance, when asking the model to autoregressively generate the completion tokens from "Babies cry because" and want to force the generation to include the word "lonely", it can induce the model to generate sequences like "Babies cry because they are lonely", as well as "Babies cry because of their loneliness".</p>
<p dir="auto">I think that this could be implemented as:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="model.generate(force_tokens=[[&quot;lonely&quot;, &quot;loneliness&quot;, ...], [&quot;happy&quot;, &quot;happiness&quot;, ...]])"><pre class="notranslate"><code class="notranslate">model.generate(force_tokens=[["lonely", "loneliness", ...], ["happy", "happiness", ...]])
</code></pre></div>
<p dir="auto">where the input to <code class="notranslate">force_tokens</code> is a 2D array, where each 1D array is a list of different forms of the desired token.</p>
<p dir="auto">Otherwise it could be:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="model.generate(force_tokens=[&quot;lonely&quot;, &quot;happy&quot;])"><pre class="notranslate"><code class="notranslate">model.generate(force_tokens=["lonely", "happy"])
</code></pre></div>
<p dir="auto">but in this case the transformers library would need to have a built-in lemmatization engine, which I think might be something better left for the practictioner to figure out (i.e. go with the first method instead).</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">Diversity in outputs from sequence generation of LMs have always been an active problem. Though usually diversity inducing methods involved some dual implementation with a VAE or modifying the training scheme, this feature would allow the practioner to induce diversity in a very <em>controllable</em> way.</p>
<p dir="auto">A large pretrained language model probably has the capacity to generate all kinds of different expressions given an input, but usually the generation gets limited to the highest probable outputs. Clearly one solution is to use sampling instead, but this goes back to the problem of controllability. This level of control is extremely useful in model implementations that aim to learn a syntactic transformations that need to preserve certain entities, or QA verbalizers where we have pre-existing knowledge of what the answer should be.</p>
<p dir="auto">Instead of making it generate a lot of sequences and filtering out for desired ones, this would allow to force it to generate an output that we want, which a large LM probably can do well; even if it can't figure out a way, then we can just filter out the low probabability outputs based on a threshold.</p>

<h2 dir="auto">Your contribution</h2>
<p dir="auto">I'm happy to submit a PR for the full implementation if there aren't any reasons to object this feature.</p>
<p dir="auto">But I do believe I should get some feedback on this idea before proceeding with an implementation, since it's not exactly clear what's the best way to introduce this functionality to the library.</p>
<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li>any env</li>
</ul>
<h3 dir="auto">Who can help</h3>
<ul dir="auto">
<li>discussed with <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/julien-c/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/julien-c">@julien-c</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a> and <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a></li>
</ul>
<h2 dir="auto">Information</h2>
<ul dir="auto">
<li>The hub will soon reject push with invalid model card metadata,</li>
<li><strong>only when <code class="notranslate">datasets</code>, <code class="notranslate">model-index</code> or <code class="notranslate">license</code> are present</strong>, their content need to follow the specification cf. <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="993215931" data-permission-text="Title is private" data-url="https://github.com/huggingface/huggingface_hub/issues/342" data-hovercard-type="pull_request" data-hovercard-url="/huggingface/huggingface_hub/pull/342/hovercard" href="https://github.com/huggingface/huggingface_hub/pull/342">huggingface/huggingface_hub#342</a></li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Train a model</li>
<li>Do not association any datasets</li>
<li>The trained model and the model card are rejected by the server</li>
</ol>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">trainer.py git push should be successfull, even with the coming patch <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="993259320" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/13514" data-hovercard-type="pull_request" data-hovercard-url="/huggingface/transformers/pull/13514/hovercard" href="https://github.com/huggingface/transformers/pull/13514">#13514</a></p>