title: Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical
    Report
summary: Multimodal Foundation Models (MMFMs) have shown remarkable performance on
  various computer vision and natural language processing tasks. However, their
  performance on particular tasks such as document understanding is still
  limited. They also require more compute, time, and engineering resources to
  finetune and deploy compared to traditional, unimodal models. In this report,
  we present Multimodal Structured Generation, a general framework which
  constrains the output logits of frozen MMFMs to force them to reason before
  responding with structured outputs that downstream APIs can parse and use. We
  provide a detailed account of our approach, including the technical details,
  theoretical discussions, and final evaluation results in the 2nd Multimodal
  Foundation Models Challenge hosted by the Computer Vision and Pattern
  Recognition (CVPR) conference. Our approach achieved the second highest score
  in the hidden test set for Phase 2 and third highest overall. This shows the
  method's ability to generalize to unseen tasks. And that simple engineering can
  beat expensive & complicated modelling steps as we first discussed in our
  paper, Retrieval Augmented Structured Generation: Business Document Information
  Extraction as Tool Use. All of our scripts, deployment steps, and evaluation
  results can be accessed in https://github.com/leloykun/MMFM-Challenge
paper_authors._id: 6670f843c4148dbf42778bc6
paper_authors.name: Franz Louis Cesista
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-18T03:07:31.274Z
paper_authors._id: 60d36cdb07da9c17c727090f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647859018354-60d36cdb07da9c17c727090f.png
paper_authors.fullname: Franz Louis Cesista
paper_authors.type: user
paper_authors.user: leloy
paper_title: Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical
    Report
paper_summary: Multimodal Foundation Models (MMFMs) have shown remarkable performance on
  various computer vision and natural language processing tasks. However, their
  performance on particular tasks such as document understanding is still
  limited. They also require more compute, time, and engineering resources to
  finetune and deploy compared to traditional, unimodal models. In this report,
  we present Multimodal Structured Generation, a general framework which
  constrains the output logits of frozen MMFMs to force them to reason before
  responding with structured outputs that downstream APIs can parse and use. We
  provide a detailed account of our approach, including the technical details,
  theoretical discussions, and final evaluation results in the 2nd Multimodal
  Foundation Models Challenge hosted by the Computer Vision and Pattern
  Recognition (CVPR) conference. Our approach achieved the second highest score
  in the hidden test set for Phase 2 and third highest overall. This shows the
  method's ability to generalize to unseen tasks. And that simple engineering can
  beat expensive & complicated modelling steps as we first discussed in our
  paper, Retrieval Augmented Structured Generation: Business Document Information
  Extraction as Tool Use. All of our scripts, deployment steps, and evaluation
  results can be accessed in https://github.com/leloykun/MMFM-Challenge
paper_ai_summary: A framework constraining multimodal foundation models to produce structured outputs enhances their performance in document understanding tasks.
paper_ai_keywords: Multimodal Foundation Models document understanding unimodal models multimodal structured generation output logits structured outputs second Multimodal Foundation Models Challenge CVPR conference Retrieval Augmented Structured Generation business document information extraction