<p dir="auto">Transformers has a Do Repeat Yourself policy in the sense that it does not provide building blocks that we then mix and match, but we strive to have each model be self-contained in terms of code, at the price of code duplication. You can find more about this philosophy in <a href="https://huggingface.co/blog/transformers-design-philosophy" rel="nofollow">this blog post</a>.</p>
<p dir="auto">There are instances in the library (mostly with older models) where this is not respected. This issue will serve as a tracker for all those instances, so that the library is cleaner and each model/tokenizer/config is easier to tweak by itself. This will also make it easier for us to test individual models in autonomy.</p>
<p dir="auto">If you wish to make a contribution to Transformers, you can help! Pick a config/model/tokenizer in the list below (double-check someone is not working on it already by searching this page!) and indicate with a comment that wish to work on it. Read our <a href="https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md">contributing guide</a>  as well as the section below, and once you are ready, open a PR and tag <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a> on it.</p>
<h2 dir="auto">How to remove a dependency from another model</h2>
<p dir="auto">There are two different types of dependencies: either a configuration/model/tokenizer uses an intermediate object from another model (example: some tokenizer uses the <code class="notranslate">BasicTokenizer</code> defined in the <code class="notranslate">tokenization_bert</code> module, or it subclasses another configuration/model/tokenizer.</p>
<p dir="auto">In the first case, the object code should just be copied inside the file, with a "Copied from" statement. This will make sure that code is always kept up to date even if the basic object is modified. For instance, if a tokenizer is using <code class="notranslate">BasicTokenizer</code>, go copy the code in <code class="notranslate">tokenization_bert</code> for that class, then paste it in the tokenizer module you are treating and add the following copied from comment:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Copied from transformers.models.bert.tokenization_bert.BasicTokenizer
class BasicTokenizer(object):
..."><pre class="notranslate"><span class="pl-c"># Copied from transformers.models.bert.tokenization_bert.BasicTokenizer</span>
<span class="pl-k">class</span> <span class="pl-v">BasicTokenizer</span>(<span class="pl-s1">object</span>):
...</pre></div>
<p dir="auto">In the second case, the code of the class (and all its building blocks) should be copied and renamed to be prefixed by the model: for instance if you are copying code from the modeling_bert module to build Roberta, you replace all <code class="notranslate">BertLayer</code>, <code class="notranslate">BertOutput</code> etc... by <code class="notranslate">RobertaLayer</code>, <code class="notranslate">RobertaOutput</code>...<br>
You should then add a copied from statement (when the copy is without any modification) like this one:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Copied from transformers.models.bert.modeling_bert.BertAttention with Bert-&gt;Roberta
class RobertaAttention(nn.Module):
..."><pre class="notranslate"><span class="pl-c"># Copied from transformers.models.bert.modeling_bert.BertAttention with Bert-&gt;Roberta</span>
<span class="pl-k">class</span> <span class="pl-v">RobertaAttention</span>(<span class="pl-s1">nn</span>.<span class="pl-v">Module</span>):
...</pre></div>
<p dir="auto">Note the replacement pattern that will adapt all names used. Note that:</p>
<ul dir="auto">
<li>you can add more of those patterns, separated by a comma like <a href="https://github.com/huggingface/transformers/blob/c28d04e9e252a1a099944e325685f14d242ecdcd/src/transformers/models/blenderbot_small/modeling_blenderbot_small.py#L1388">here</a>.</li>
<li>you can ask to replace all possible casings like <a href="https://github.com/huggingface/transformers/blob/c28d04e9e252a1a099944e325685f14d242ecdcd/src/transformers/models/mobilebert/modeling_mobilebert.py#L1549">here</a></li>
<li>you can just copy one method and not the whole class like <a href="https://github.com/huggingface/transformers/blob/c28d04e9e252a1a099944e325685f14d242ecdcd/src/transformers/models/roberta/modeling_roberta.py#L741">here</a></li>
</ul>
<p dir="auto"><strong>NB:</strong> No need for copied from statements in the config (the defaults are probably different anyway).</p>
<h2 dir="auto">Objects to cover</h2>
<h3 dir="auto">Configurations</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Flaubert config (should not use XLM)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> LayoutLM config (should not use Bert)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> LongformerConfig (should not use Roberta)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> MarkupLMConfig (should not Roberta)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> RobertaConfig (should not use Bert)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> XLM-ProphetNet config (should not use ProphetNet)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> XLM-Roberta config (should not use Roberta)</li>
</ul>
<h3 dir="auto">Models</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> BertGeneration (should not use BertEncoder)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Camembert (should not use Roberta) (PyTorch + TF)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Flaubert (should not use XLM) (PyTorch + TF)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> mT5: <del>PyTorch</del>, TensorFlow, Flax (should not use T5)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> XLM-ProphetNet (should not use ProphetNet)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Xlm-Roberta: <del>PyTorch</del>, TensorFlow, Flax (should not use Roberta)</li>
</ul>
<h3 dir="auto">Tokenizers</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> BertJapanese (should not use any imports from tokenization bert)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Blenderbot (should not use Roberta) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Clip (should not use BasicTokenizer from Bert)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> ConvBERT (should not use Bert) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Cpm tokenizer (should not use XLNet) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Derberta tokenizer (should not use GPT2) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> DistilBert (should not use Bert) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Electra (should not use Bert) (fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Flaubert (should not use XLM)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Funnel (should not use Bert) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Herbert (should not BasicTokenizer from Bert and XLM)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> LayoutLM (should not use Bert) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> LED (should not use BART) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Longformer (should not use Roberta) (fast tokenizer)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Luke (should not use Roberta)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Lxmert (should not use Bert) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> MobileBert (should not use Bert) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Openai-GPT (should not use BasicTokenizer from Bert)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> ProphetNet (should not use BasicTokenzier and WordPieceTokenizer from Bert)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Retribert tokenizer (should not use Bert) (slow/fast)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Roformer tokenizer (should not use any imports from tokenization bert)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Squeezebert tokenizer (should not use Bert) (slow/fast)</li>
</ul>
<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature request</h1>
<p dir="auto">Over at EleutherAI we've recently released a 20 billion parameter autoregressive gpt model (see <a href="https://github.com/EleutherAI/gpt-neox">gpt-neox</a> for a link to the weights). It would be great to get this into transformers!</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">gpt-neox library is not quite as user-friendly as transformers, and is designed for efficient large scale training over inference. So we think integrating it into transformers would be great for accessibility.</p>
<h2 dir="auto">Your contribution</h2>
<p dir="auto">Personally, I can dedicate a bit of time into integrating this model. We already have a <a href="https://github.com/EleutherAI/gpt-neox/pull/480" data-hovercard-type="pull_request" data-hovercard-url="/EleutherAI/gpt-neox/pull/480/hovercard">PR to convert the weights to HF format</a>, although:</p>
<ol dir="auto">
<li>I suspect you would want to introduce a new model class for it.</li>
<li>It is not merged / thoroughly tested with all possible configurations yet. GPT-NeoX has a bunch of configuration options, and it might be more straightforward to focus on <em>just</em> introducing a model class for GPT-NeoX-20B (which should largely be similar to GPT-J, with some caveats, see next section)</li>
</ol>
<h2 dir="auto">Difficulties</h2>
<ul dir="auto">
<li>Whilst we do have a script to merge <a href="https://github.com/EleutherAI/gpt-neox/pull/466" data-hovercard-type="pull_request" data-hovercard-url="/EleutherAI/gpt-neox/pull/466/hovercard">model parallel checkpoints</a> that will be merged soon, in larger models we see some performance loss when merging mp ranks, due to there being some very slight differences in the replicated parameters between ranks (see the thread above for more details). If we want to integrate GPT-NeoX-20B as a model to be used on a single GPU, we need to figure out how to address this. I'm not sure if this bug is specific to neox, or was introduced in megatron or deepspeed, but I believe the bigscience team <a href="https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/239" data-hovercard-type="pull_request" data-hovercard-url="/bigscience-workshop/Megatron-DeepSpeed/pull/239/hovercard">are also looking at merging mp models</a>, so I guess we'll find out soon.</li>
<li>If we do integrate the model without model parallelism - it will be too large to run on most consumer GPUs. During inference, it takes ~45GB of GPU memory to run, and during training much more.</li>
</ul>