readme: ---
  license: mit
  tags:
  - generated_from_trainer
  datasets:
  - funsd-layoutlmv3
  model-index:
  - name: lilt-en-funsd
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # lilt-en-funsd
  
  This model is a fine-tuned version of [SCUT-DLVCLab/lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-roberta-en-base) on the funsd-layoutlmv3 dataset.
  It achieves the following results on the evaluation set:
  - Loss: 1.6117
  - Answer: {'precision': 0.8821428571428571, 'recall': 0.9069767441860465, 'f1': 0.8943874471937237, 'number': 817}
  - Header: {'precision': 0.6126126126126126, 'recall': 0.5714285714285714, 'f1': 0.591304347826087, 'number': 119}
  - Question: {'precision': 0.9045045045045045, 'recall': 0.9322191272051996, 'f1': 0.9181527206218564, 'number': 1077}
  - Overall Precision: 0.8797
  - Overall Recall: 0.9006
  - Overall F1: 0.8900
  - Overall Accuracy: 0.8204
  
  ## Model Usage
  
  ```python
  from transformers import LiltForTokenClassification, LayoutLMv3Processor
  from PIL import Image, ImageDraw, ImageFont
  import torch
  
  # load model and processor from huggingface hub
  model = LiltForTokenClassification.from_pretrained("philschmid/lilt-en-funsd")
  processor = LayoutLMv3Processor.from_pretrained("philschmid/lilt-en-funsd")
  
  
  # helper function to unnormalize bboxes for drawing onto the image
  def unnormalize_box(bbox, width, height):
      return [
          width * (bbox[0] / 1000),
          height * (bbox[1] / 1000),
          width * (bbox[2] / 1000),
          height * (bbox[3] / 1000),
      ]
  
  
  label2color = {
      "B-HEADER": "blue",
      "B-QUESTION": "red",
      "B-ANSWER": "green",
      "I-HEADER": "blue",
      "I-QUESTION": "red",
      "I-ANSWER": "green",
  }
  # draw results onto the image
  def draw_boxes(image, boxes, predictions):
      width, height = image.size
      normalizes_boxes = [unnormalize_box(box, width, height) for box in boxes]
  
      # draw predictions over the image
      draw = ImageDraw.Draw(image)
      font = ImageFont.load_default()
      for prediction, box in zip(predictions, normalizes_boxes):
          if prediction == "O":
              continue
          draw.rectangle(box, outline="black")
          draw.rectangle(box, outline=label2color[prediction])
          draw.text((box[0] + 10, box[1] - 10), text=prediction, fill=label2color[prediction], font=font)
      return image
  
  
  # run inference
  def run_inference(image, model=model, processor=processor, output_image=True):
      # create model input
      encoding = processor(image, return_tensors="pt")
      del encoding["pixel_values"]
      # run inference
      outputs = model(**encoding)
      predictions = outputs.logits.argmax(-1).squeeze().tolist()
      # get labels
      labels = [model.config.id2label[prediction] for prediction in predictions]
      if output_image:
          return draw_boxes(image, encoding["bbox"][0], labels)
      else:
          return labels
  
  
  run_inference(dataset["test"][34]["image"])
  
  ```
  
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 5e-05
  - train_batch_size: 8
  - eval_batch_size: 8
  - seed: 42
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - training_steps: 2500
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch  | Step | Validation Loss | Answer                                                                                                   | Header                                                                                                   | Question                                                                                                  | Overall Precision | Overall Recall | Overall F1 | Overall Accuracy |
  |:-------------:|:------:|:----:|:---------------:|:--------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------:|:-----------------:|:--------------:|:----------:|:----------------:|
  | 0.0211        | 10.53  | 200  | 1.5528          | {'precision': 0.8458904109589042, 'recall': 0.9069767441860465, 'f1': 0.8753691671588896, 'number': 817} | {'precision': 0.5684210526315789, 'recall': 0.453781512605042, 'f1': 0.5046728971962617, 'number': 119}  | {'precision': 0.896551724137931, 'recall': 0.89322191272052, 'f1': 0.8948837209302325, 'number': 1077}    | 0.8596            | 0.8728         | 0.8662     | 0.8011           |
  | 0.0132        | 21.05  | 400  | 1.3143          | {'precision': 0.8447058823529412, 'recall': 0.8788249694002448, 'f1': 0.8614277144571085, 'number': 817} | {'precision': 0.6020408163265306, 'recall': 0.4957983193277311, 'f1': 0.543778801843318, 'number': 119}  | {'precisio