<h3 dir="auto">System Info</h3>
<p dir="auto">(Possible duplicate: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="810994180" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/10256" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/10256/hovercard" href="https://github.com/huggingface/transformers/issues/10256">#10256</a>)<br>
I have written a custom tokenizer that builds on top of <code class="notranslate">BertTokenizer</code> (returns one extra list of ids that will later be embedded in a custom model). I have pushed it to Hub as well. Now, how can I allow others to use it? The code for the tokenizer is uploaded to Hub along with the code for the model (they are in the same file), but since I cannot register the tokenizer with <code class="notranslate">AutoTokenizer</code> like I can do for models (<code class="notranslate">CustomModel.register_for_auto_class("AutoModel")</code>), others cannot load this tokenizer, and hence use the model.</p>
<p dir="auto">Is there a workaround for this?</p>
<p dir="auto">Version: 4.27.4</p>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ArthurZucker/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ArthurZucker">@ArthurZucker</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/younesbelkada/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/younesbelkada">@younesbelkada</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">The code for both the tokenizer and model can be found here: <a href="https://huggingface.co/mcgill-babylm/bert_ds10M_np512_nh2_nl2_hs128_postags_ungrouped/blob/main/pos_bert.py" rel="nofollow">https://huggingface.co/mcgill-babylm/bert_ds10M_np512_nh2_nl2_hs128_postags_ungrouped/blob/main/pos_bert.py</a></p>
<p dir="auto">I am able to load the model with no problems since I push it after registering it as follows</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="BertForMaskedLMWithPOSEmb.register_for_auto_class(&quot;AutoModel&quot;)
BertForMaskedLMWithPOSEmb.register_for_auto_class(&quot;AutoModelForMaskedLM&quot;)"><pre class="notranslate"><code class="notranslate">BertForMaskedLMWithPOSEmb.register_for_auto_class("AutoModel")
BertForMaskedLMWithPOSEmb.register_for_auto_class("AutoModelForMaskedLM")
</code></pre></div>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">I should be able to register custom tokenizers with <code class="notranslate">AutoTokenizer</code> (which might be a new feature request) or work around it somehow to allow other users to use a custom tokenizer.</p>
<h3 dir="auto">Model description</h3>
<p dir="auto">I would like to implement a new model architecture.</p>
<h2 dir="auto">Short description</h2>
<p dir="auto">RWKV v2 is an "RNN with transformer-level performance, without using attention. Similar to Apple's Attention Free Transformer. All trained models open-source. Inference is very fast (even on CPUs) and might work on cell phones. There's also a GPT-type implementation." -- (<a href="https://twitter.com/HochreiterSepp/status/1524270961314484227" rel="nofollow">Hochreiter's description</a>)</p>
<p dir="auto">RWKV v2 is parallelizable because the time-decay of each channel is data-independent (and trainable). For example, in usual RNN you can adjust the time-decay of a channel from say 0.8 to 0.5 (these are called "gates"), while in RWKV v2 you simply move the information from a W-0.8-channel to a W-0.5-channel to achieve the same effect. RWKV can leverage GPUs, but doesn't need to.</p>
<h3 dir="auto">Open source status</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> The model implementation is available</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> The model weights are available</li>
</ul>
<h3 dir="auto">Provide useful links for the implementation</h3>
<h2 dir="auto">Implementation and weights</h2>
<p dir="auto">There's an implementation at <a href="https://github.com/BlinkDL/RWKV-LM">BlinkDL/RWKV-LM</a> which also gives a detailed description of the model internals and some performance benchmarks. Model weights currently are being trained for a few datasets, including the Pile (see e.g. <a href="https://github.com/BlinkDL/RWKV-v2-RNN-Pile/">BlinkDL/RWKV-v2-RNN-Pile</a>) and <a href="https://gigaword.dk" rel="nofollow">Danish Gigaword</a> by me. Both will be openly available - some checkpoints for the Pile already are, even though it's an ongoing process.</p>
<h2 dir="auto">Status</h2>
<p dir="auto">The model seems quite exciting and I'm able to replicate preliminary results. I'm already talking with <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/BlinkDL/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/BlinkDL">@BlinkDL</a> about the implementation. I'm happy to implement/port the model architecture (for both RNN and GPT variants), tokenizer, and tests myself (and have already started) and would appreciate help and advice.</p>