LayoutLMV3 Training with Morethan 512 tokens.
Hi, It is mentioned in the Research paper and in Model docs that the model is trained with `maximum_length = 512` tokens. But when we have large tokens with an average of 1124 tokens, I thought to increase `maximum_length = 1024`. But since it is limited, I have used `Stride` and implemented a dataclass.

![image](https://user-images.githubusercontent.com/112476954/192162483-26a375c1-9584-4122-865e-069e6f304332.png)

So when I have a long text sentence with 900 in length in this example. I got
![image](https://user-images.githubusercontent.com/112476954/192162557-9efe39a4-6915-4df5-9995-c34f2bce36d1.png)

But when the same sent to trainer
![image](https://user-images.githubusercontent.com/112476954/192162648-12599fad-9f4c-4bbc-94e7-13fbf5009162.png)

I got error like below:
![image](https://user-images.githubusercontent.com/112476954/192162684-b629bcaa-4b06-477b-9f20-4e7fa34c00c3.png)

From the Error, I understood that My `input_ids` are in the shape of [2,512] and once after going into the trainer, they become [1,2,512] after batch size is added. 
But the trainer is expecting My `input_ids` to be in [512] after squeezed, so the trainer adds batch size 1 to make [1,512].

Now how to send my datasamples of length more than 512, with stride in processor, though i am able to load the data. But the trainer is not expecting sizes more than 512. Any suggestions here

Thank you

cc @NielsRogge <|||||>> 
@purnasai-cyient 
Hi,
I have same issue with you. Have you solved it?

> Hi, It is mentioned in the Research paper and in Model docs that the model is trained with `maximum_length = 512` tokens. But when we have large tokens with an average of 1124 tokens, I thought to increase `maximum_length = 1024`. But since it is limited, I have used `Stride` and implemented a dataclass.
> 
> ![image](https://user-images.githubusercontent.com/112476954/192162483-26a375c1-9584-4122-865e-069e6f304332.png)
> 
> So when I have a long text sentence with 900 in length in this example. I got ![image](https://user-images.githubusercontent.com/112476954/192162557-9efe39a4-6915-4df5-9995-c34f2bce36d1.png)
> 
> But when the same sent to trainer ![image](https://user-images.githubusercontent.com/112476954/192162648-12599fad-9f4c-4bbc-94e7-13fbf5009162.png)
> 
> I got error like below: ![image](https://user-images.githubusercontent.com/112476954/192162684-b629bcaa-4b06-477b-9f20-4e7fa34c00c3.png)
> 
> From the Error, I understood that My `input_ids` are in the shape of [2,512] and once after going into the trainer, they become [1,2,512] after batch size is added. But the trainer is expecting My `input_ids` to be in [512] after squeezed, so the trainer adds batch size 1 to make [1,512].
> 
> Now how to send my datasamples of length more than 512, with stride in processor, though i am able to load the data. But the trainer is not expecting sizes more than 512. Any suggestions here
> 
> Thank you

<|||||>Hello @purnasai-cyient and everyone! 

I had the same issue with LayoutLMv3 and because I think this problem is common for document information extraction task so I will describe how I dealt with that:

### 1. Training:

As you may know, first of all we have to change configurations of processor by using `stride` and `padding` and `offset_mapping`:

```
....
......
processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=False)

encoding = processor(images, words, boxes=boxes, word_labels=word_labels, truncation=True, stride =128, 
         padding="max_length", max_length=512, return_overflowing_tokens=True, return_offsets_mapping=True)

offset_mapping = encoding.pop('offset_mapping')

overflow_to_sample_mapping = encoding.pop('overflow_to_sample_mapping')
```

I'm not completely sure why should we turn offset_mapping True while we have to pop it after encoding but I think it's necessary to do that. To clarify, if you followed the [NielsRogge](https://github.com/NielsRogge)'s notebook ([you can find the notebook here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb)), we have to change the `prepare_examples` method like this:

```
def prepare_examples(examples):
  images = [Image.open(path).convert("RGB") for path in examples['image_path']] 
  words = examples[text_column_name]
  boxes = examples[boxes_column_name]
  word_labels = examples[label_column_name]
  encoding = processor(images, words, boxes=boxes, word_labels=word_labels, truncation=True, stride =128, 
         padding="max_length", max_length=512, return_overflowing_tokens=True, return_offsets_mapping=True)  offset_mapping = encoding.pop('offset_mapping')
  overflow_to_sample_mapping = encoding.pop('overflow_to_sample_mapping')
  return encoding
```
next you have to follow the steps normally, without any changes and train the model!

Note: It's completely normal if the number of rows in your dataset after `mapping` doesn't match with to number of your data. It's due to that now if we pass 512 tokens for a document, we would store the next tokens in another row of data (from token 512 to 1024, next from 1025 to 1536 and...)

### 2. Inference:

Inference section is a little bit tricky. Again I will describe the setups based on [mentioned notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb). Obviously we have to implement our processor with `stride` and `padding` like training phase. 

```
encoding = processor(images, words, boxes=boxes, word_labels=word_labels, truncation=True, stride =128, 
         padding="max_length", max_length=512, return_overflowing_tokens=True, return_offsets_mapping=True)

offset_mapping = encoding.pop('offset_mapping')

overflow_to_sample_mapping = encoding.pop('overflow_to_sample_mapping')
```

Next, we have to change the shape of encoding to handle multiple pages (as I said in part 1, we divided large tokens to separate entities, so now the results are 2D. I have shaped them in this way:

```
# change the shape of pixel values
x = []
for i in range(0, len(encoding['pixel_values'])):
     x.append(encoding['pixel_values'][i])
x = torch.stack(x)
encoding['pixel_values'] = x
```
so if we print encoding items, we will have something like this:

```
for k,v in encoding.items():
  print(k,v.shape)
```
results:
```
input_ids torch.Size([3, 512])
attention_mask torch.Size([3, 512])
bbox torch.Size([3, 512, 4])
pixel_values torch.Size([3, 3, 224, 224])
```
As we can see, in my case, document is divided to 3 parts (for example size of `input_ids` is [3,512] or 3x512 which if we use normal processing, we will get just one array [1, 512] for all the cases). So we're doing fine till now. We have to pass encoding to the model to get the predictions:

```
with torch.no_grad():
  outputs = model(**encoding)

# The model outputs logits of shape (batch_size, seq_len, num_labels).
logits = outputs.logits
print(logits.shape)

# We take the highest score for each token, using argmax. This serves as the predicted label for each token.
predictions = logits.argmax(-1).squeeze().tolist()
token_boxes = encoding.bbox.squeeze().tolist()

if (len(token_boxes) == 512):
  predictions = [predictions]
  token_boxes = [token_boxes]
```
Last lines (if clause) in code above, is because in cases that the number of tokens are less than 512, we will get 1D array, we have to put them in a list to prevent errors for next step. 

Finally, now we have `predictions` and `token_boxes` from the model, you can also reach the text of each bbox by using:
`processor.tokenizer.decode(encoding["input_ids"][i][j])` which` i `and` j `corresponds to the entity that you want to extract the text of it. Just as an example, we could find predictions by traversing `token_boxes` with for loop (you could do whatever you want, because we needed predictions and bboxes and we have them now! processing them is up to you ;) )

```
# this is just an example, change this code for your project!
for i in range(0, len(token_boxes)):
      for j in range(0, len(token_boxes[i])):
             print("label is: {}, bbox is: {} and the text is: {}".format(predictions[i][j], 
                    token_boxes[i][j],  processor.tokenizer.decode(encoding["input_ids"][i][j]) )
             
```


<|||||>Hi @alitavanaali,

Thank you for sharing your solution - I will try it out soon.

One question, if you don't mind sharing your thoughts: if you break a document into 3 chunks, wouldn't the prediction of the last text (in chunk 3) in the sequence miss the context from the very first text (in chunk 1), thus, degrading the performance of the model for later chunks? In other words, since the self-attention is still of shape 512 x 512 and information is not shared across chunks, is chunking a document a good idea?<|||||>Hello, 
Thank you for the solution @alitavanaali . I have several questions:

- The inference solution does only work for one image right? since after the encoding, the x is stacked together so we loose the index of every image. My question is how to make it work for a batch inference? while having as output the predicted labels by image?
- In theory, Layoutlm is trained on Publaynet dataset which has a lot of documents with a lot of text (most of them with more than 512 tokens). How so that Layoutlmv3 is limited to 512 and thus for so little text in image?<|||||>I'm leaving this comment in case it may help others. If you follow @alitavanaali's approach, and you wish to align back to the original text (by combining the strided window regions together), I do the following:
```
true_predictions =[]
true_boxes = []
STRIDE_COUNT = 128
for i, (pred, box, mapped) in enumerate(zip(predictions, token_boxes, offset_mapping)):
    is_subword = np.array(mapped.squeeze().tolist())[:,0] != 0
    if i == 0:
        true_predictions += [id2label[pred_] for idx, pred_ in enumerate(pred) if (not is_subword[idx])]
        true_boxes += [unnormalize_box(box_, width, height) for idx, box_ in enumerate(box) if not is_subword[idx]]
    else:
        true_predictions += [id2label[pred_] for idx, pred_ in enumerate(pred) if (not is_subword[idx])][STRIDE_COUNT - sum(is_subword[:STRIDE_COUNT]):]
        true_boxes += [unnormalize_box(box_, width, height) for idx, box_ in enumerate(box) if not is_subword[idx]][STRIDE_COUNT - sum(is_subword[:STRIDE_COUNT]):]
```