<h3 dir="auto">Model description</h3>
<p dir="auto">LayoutLMv3 is a pre-trained multimodal Transformer for Document AI with unified text and image masking. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model. For example, LayoutLMv3 can be fine-tuned for both text-centric tasks, including form understanding, receipt understanding, and document visual question answering, and image-centric tasks such as document image classification and document layout analysis.</p>
<p dir="auto">LayoutLMv3 greatly simplifies training and reduces the number of parameters compared to v3, making it an important milestone in document understanding.</p>
<p dir="auto"><a href="https://arxiv.org/abs/2204.08387" rel="nofollow">LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking</a> Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei, Preprint 2022.</p>
<h3 dir="auto">Open source status</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> The model implementation is available</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> The model weights are available</li>
</ul>
<h3 dir="auto">Provide useful links for the implementation</h3>
<p dir="auto"><a href="https://huggingface.co/microsoft/layoutlmv3-base" rel="nofollow">Huggingface Pretrained Download</a></p>
<h2 dir="auto">❓ Questions &amp; Help</h2>

<p dir="auto">The BertForQuestionAnswering sample code creates duplicate [CLS] tokens. Wondering why:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
question, text = &quot;Who was Jim Henson?&quot;, &quot;Jim Henson was a nice puppet&quot;
input_text = &quot;[CLS] &quot; + question + &quot; [SEP] &quot; + text + &quot; [SEP]&quot;
input_ids = tokenizer.encode(input_text)
token_type_ids = [0 if i &lt;= input_ids.index(102) else 1 for i in range(len(input_ids))]
start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
# a nice puppet
tokenizer.decode(input_ids)
#'[CLS] [CLS] who was jim henson? [SEP] jim henson was a nice puppet [SEP] [SEP]'  "><pre class="notranslate"><code class="notranslate">tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
input_text = "[CLS] " + question + " [SEP] " + text + " [SEP]"
input_ids = tokenizer.encode(input_text)
token_type_ids = [0 if i &lt;= input_ids.index(102) else 1 for i in range(len(input_ids))]
start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
# a nice puppet
tokenizer.decode(input_ids)
#'[CLS] [CLS] who was jim henson? [SEP] jim henson was a nice puppet [SEP] [SEP]'  
</code></pre></div>
<p dir="auto">If I remove the extra [CLS], the extraction doesn't work. It's exactly two tokens off:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="input_ids = tokenizer.encode(input_text, add_special_tokens=False)
...rerun same code as above...
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
# was a"><pre class="notranslate"><code class="notranslate">input_ids = tokenizer.encode(input_text, add_special_tokens=False)
...rerun same code as above...
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
# was a
</code></pre></div>
<p dir="auto">What am I doing wrong? How can I get the extraction working without duplicate [CLS] tokens? (and duplicate final [SEP] tokens BTW).</p>
<p dir="auto">The sample code comes right from the docs: <a href="https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering" rel="nofollow">https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering</a></p>