title: BigDocs: An Open and Permissively-Licensed Dataset for Training
    Multimodal Models on Document and Code Tasks
summary: Multimodal AI has the potential to significantly enhance
  document-understanding tasks, such as processing receipts, understanding
  workflows, extracting data from documents, and summarizing reports. Code
  generation tasks that require long-structured outputs can also be enhanced by
  multimodality. Despite this, their use in commercial applications is often
  limited due to limited access to training data and restrictive licensing, which
  hinders open access. To address these limitations, we introduce BigDocs-7.5M, a
  high-quality, open-access dataset comprising 7.5 million multimodal documents
  across 30 tasks. We use an efficient data curation process to ensure our data
  is high-quality and license-permissive. Our process emphasizes accountability,
  responsibility, and transparency through filtering rules, traceable metadata,
  and careful content analysis. Additionally, we introduce BigDocs-Bench, a
  benchmark suite with 10 novel tasks where we create datasets that reflect
  real-world use cases involving reasoning over Graphical User Interfaces (GUI)
  and code generation from images. Our experiments show that training with
  BigDocs-Bench improves average performance up to 25.8% over closed-source
  GPT-4o in document reasoning and structured output tasks such as
  Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a
  preference for outputs from models trained on BigDocs over GPT-4o. This
  suggests that BigDocs can help both academics and the open-source community
  utilize and improve AI tools to enhance multimodal capabilities and document
  reasoning. The project is hosted at https://bigdocs.github.io .
paper_authors._id: 675714ab0ab5c19744cedc4c
paper_authors.name: Juan Rodriguez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:05:01.009Z
paper_authors._id: 63a614d264f470027818b066
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a614d264f470027818b066/Q5Eih2VqD4NaHB_MkNp32.jpeg
paper_authors.fullname: Juan A. Rodriguez
paper_authors.type: user
paper_authors.user: joanrodai
paper_authors._id: 675714ab0ab5c19744cedc4d
paper_authors.name: Xiangru Jian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:49:25.747Z
paper_authors._id: 66155491f1214e73d69074b5
paper_authors.avatarUrl: /avatars/00572ab695a4188422e8ee38fc87680b.svg
paper_authors.fullname: Xiangru Jian
paper_authors.type: user
paper_authors.user: EdwardXJ
paper_authors._id: 675714ab0ab5c19744cedc4e
paper_authors.name: Siba Smarak Panigrahi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:22:47.380Z
paper_authors._id: 67574a29dd1f47442293becd
paper_authors.avatarUrl: /avatars/5e6a28899e5ea631d8f0677a9dd1a51a.svg
paper_authors.fullname: Siba Smarak Panigrahi
paper_authors.type: user
paper_authors.user: sibasmarakp
paper_authors._id: 675714ab0ab5c19744cedc4f
paper_authors.name: Tianyu Zhang
paper_authors._id: 675714ab0ab5c19744cedc50
paper_authors.name: Aarash Feizi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:51:34.277Z
paper_authors._id: 62e98e784fa4bc6de6c0f65b
paper_authors.avatarUrl: /avatars/4354e022061d78ec26b14c0b0cd4e433.svg
paper_authors.fullname: Aarash Feizi
paper_authors.type: user
paper_authors.user: aarashfeizi
paper_authors._id: 675714ab0ab5c19744cedc51
paper_authors.name: Abhay Puri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:51:41.042Z
paper_authors._id: 65830af2a1707aa10effcc32
paper_authors.avatarUrl: /avatars/0626454399b711fca7fb2b66fcecaca8.svg
paper_authors.fullname: Abhay Puri
paper_authors.type: user
paper_authors.user: abhaypuri
paper_authors._id: 675714ab0ab5c19744cedc52
paper_authors.name: Akshay Kalkunte
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:51:47.592Z
paper_authors._id: 647792b1e5ddd7d2a1714550
paper_authors.avatarUrl: /avatars/067cad5fef442146217fd11316985486.svg
paper_authors.fullname: Akshay Kalkunte
paper_authors.type: user
paper_authors.user: akshaykalkunte
paper_authors._id: 675714ab0ab5c19744cedc53
paper_authors.name: François Savard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:51:53.902Z
paper_authors._id: 66c600c28991d6123dd89abd
paper_authors.avatarUrl: /avatars/4236d4637a0f23e90aa5c79fa8ced466.svg
paper_authors.fullname: François Savard
paper_authors.type: user
paper_authors.user: fsavard-eai
paper_authors._id: 675714ab0ab5c19744cedc54
paper_authors.name: Ahmed Masry
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-10T10:25:39.506Z
paper_authors._id: 63efd75a5c2ceb16fc6e98fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63efd75a5c2ceb16fc6e98fc/qoA4LKuLTEr7hx90i90UK.jpeg
paper_authors.fullname: Ahmed Masry
paper_authors.type: user
paper_authors.user: ahmed-masry
paper_authors._id: 675714ab0ab5c19744cedc55
paper_authors.name: Shravan Nayak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:51:59.460Z
paper_authors._id: 643ee7606d5be535d28034f6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643ee7606d5be535d28034f6/1F241mP_buW74byC2GAWH.jpeg
paper_authors.fullname: Shravan Nayak
paper_authors.type: user
paper_authors.user: BAJUKA
paper_authors._id: 675714ab0ab5c19744cedc56
paper_authors.name: Rabiul Awal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:52:13.031Z
paper_authors._id: 60edf5e03203a5daf7d3912e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60edf5e03203a5daf7d3912e/F3PpWaZf78lsa0ZdkjzTO.jpeg
paper_authors.fullname: Rabiul Awal
paper_authors.type: user
paper_authors.user: rabiulawal
paper_authors._id: 675714ab0ab5c19744cedc57
paper_authors.name: Mahsa Massoud
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:52:43.526Z
paper_authors._id: 655502305f7b57f585042408
paper_authors.avatarUrl: /avatars/8ce57b8193ee2e6fae42d66aeaafef12.svg
paper_authors.fullname: Mahsa
paper_authors.type: user
paper_authors.user: mahsamassoud
paper_authors._id: 675714ab0ab5c19744cedc58
paper_authors.name: Amirhossein Abaskohi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:52:49.625Z
paper_authors._id: 60e32baedc56466240084155
paper_authors.avatarUrl: /avatars/a1edb3dbe5e462f9c13c79a703649aca.svg
paper_authors.fullname: Amirhossein Abaskohi
paper_authors.type: user
paper_authors.user: AmirhosseinAbaskohi
paper_authors._id: 675714ab0ab5c19744cedc59
paper_authors.name: Zichao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:53:02.483Z
paper_authors._id: 6419632bb4adb0e101aefd21
paper_authors.avatarUrl: /avatars/82ea8d77fd1d41c8a6d3a14547d076f3.svg
paper_authors.fullname: lizichao
paper_authors.type: user
paper_authors.user: lazyc
paper_authors._id: 675714ab0ab5c19744cedc5a
paper_authors.name: Suyuchen Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:53:08.933Z
paper_authors._id: 62bb1e0f3ff437e49a3088e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/bcUQmH8tKfI6DIWH9IcYp.jpeg
paper_authors.fullname: Suyuchen Wang
paper_authors.type: user
paper_authors.user: sheryc
paper_authors._id: 675714ab0ab5c19744cedc5b
paper_authors.name: Pierre-André Noël
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:53:15.480Z
paper_authors._id: 646cc16b94eb019a96e1fb2e
paper_authors.avatarUrl: /avatars/31f168a6c1ec45eb0c784d9119c1b9bf.svg
paper_authors.fullname: Pierre-Andre Noel
paper_authors.type: user
paper_authors.user: PierreAndreNoel
paper_authors._id: 675714ab0ab5c19744cedc5c
paper_authors.name: Mats Leon Richter
paper_authors._id: 675714ab0ab5c19744cedc5d
paper_authors.name: Saverio Vadacchino
paper_authors._id: 675714ab0ab5c19744cedc5e
paper_authors.name: Shubbam Agarwal
paper_authors._id: 675714ab0ab5c19744cedc5f
paper_authors.name: Sanket Biswas
paper_authors._id: 675714ab0ab5c19744cedc60
paper_authors.name: Sara Shanian
paper_authors._id: 675714ab0ab5c19744cedc61
paper_authors.name: Ying Zhang
paper_authors._id: 675714ab0ab5c19744cedc62
paper_authors.name: Noah Bolger
paper_authors._id: 675714ab0ab5c19744cedc63
paper_authors.name: Kurt MacDonald
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:53:54.337Z
paper_authors._id: 64dbefc6176ca2c9a2d799ac
paper_authors.avatarUrl: /avatars/d8f6d730091dfa0857275e48970770c6.svg
paper_authors.fullname: Kurt MacDonald
paper_authors.type: user
paper_authors.user: K-NOW
paper_authors._id: 675714ab0ab5c19744cedc64
paper_authors.name: Simon Fauvel
paper_authors._id: 675714ab0ab5c19744cedc65
paper_authors.name: Sathwik Tejaswi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T12:56:37.773Z
paper_authors._id: 63d3095c2727d7888cbb54e2
paper_authors.avatarUrl: /avatars/51fd37f4216eec309cd439e56626d6ad.svg
paper_authors.fullname: Sathwik Tejaswi Madhusudhan
paper_authors.type: user
paper_authors.user: stm4
paper_authors._id: 675714ab0ab5c19744cedc66
paper_authors.name: Srinivas Sunkara
paper_authors._id: 675714ab0ab5c19744cedc67
paper_authors.name: Joao Monteiro
paper_authors._id: 675714ab0ab5c19744cedc68
paper_authors.name: Krishnamurthy DJ Dvijotham
paper_authors._id: 675714ab0ab5c19744cedc69
paper_authors.name: Torsten Scholak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:54:13.721Z
paper_authors._id: 60ecaa5efee13fee7ada7af4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1637694419284-60ecaa5efee13fee7ada7af4.jpeg
paper_authors.fullname: Torsten Scholak
paper_authors.type: user
paper_authors.user: tscholak
paper_authors._id: 675714ab0ab5c19744cedc6a
paper_authors.name: Nicolas Chapados
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:55:42.051Z
paper_authors._id: 631f54aa5ba8c026340b13cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
paper_authors.fullname: Nicolas Chapados
paper_authors.type: user
paper_authors.user: nicolaschapados
paper_authors._id: 675714ab0ab5c19744cedc6b
paper_authors.name: Sepideh Kharagani
paper_authors._id: 675714ab0ab5c19744cedc6c
paper_authors.name: Sean Hughes
paper_authors._id: 675714ab0ab5c19744cedc6d
paper_authors.name: M. Özsu
paper_authors._id: 675714ab0ab5c19744cedc6e
paper_authors.name: Siva Reddy
paper_authors._id: 675714ab0ab5c19744cedc6f
paper_authors.name: Marco Pedersoli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:54:40.883Z
paper_authors._id: 64b829042fccad9f5ff20cc7
paper_authors.avatarUrl: /avatars/92cca111ddd0db5d998615c2257a0894.svg
paper_authors.fullname: Marco Pedersoli
paper_authors.type: user
paper_authors.user: Marcopede
paper_authors._id: 675714ab0ab5c19744cedc70
paper_authors.name: Yoshua Bengio
paper_authors._id: 675714ab0ab5c19744cedc71
paper_authors.name: Christopher Pal
paper_authors._id: 675714ab0ab5c19744cedc72
paper_authors.name: Issam Laradji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:55:29.825Z
paper_authors._id: 64062855692855e65ae31688
paper_authors.avatarUrl: /avatars/e35d22a037b8b35422d3ee982f133076.svg
paper_authors.fullname: Issam Laradji
paper_authors.type: user
paper_authors.user: issamlaradji
paper_authors._id: 675714ab0ab5c19744cedc73
paper_authors.name: Spandanna Gella
paper_authors._id: 675714ab0ab5c19744cedc74
paper_authors.name: Perouz Taslakian
paper_authors._id: 675714ab0ab5c19744cedc75
paper_authors.name: David Vazquez
paper_authors._id: 675714ab0ab5c19744cedc76
paper_authors.name: Sai Rajeswar
paper_title: BigDocs: An Open and Permissively-Licensed Dataset for Training
    Multimodal Models on Document and Code Tasks
paper_summary: Multimodal AI has the potential to significantly enhance
  document-understanding tasks, such as processing receipts, understanding
  workflows, extracting data from documents, and summarizing reports. Code
  generation tasks that require long-structured outputs can also be enhanced by
  multimodality. Despite this, their use in commercial applications is often
  limited due to limited access to training data and restrictive licensing, which
  hinders open access. To address these limitations, we introduce BigDocs-7.5M, a
  high-quality, open-access dataset comprising 7.5 million multimodal documents
  across 30 tasks. We use an efficient data curation process to ensure our data
  is high-quality and license-permissive. Our process emphasizes accountability,
  responsibility, and transparency through filtering rules, traceable metadata,
  and careful content analysis. Additionally, we introduce BigDocs-Bench, a
  benchmark suite with 10 novel tasks where we create datasets that reflect
  real-world use cases involving reasoning over Graphical User Interfaces (GUI)
  and code generation from images. Our experiments show that training with
  BigDocs-Bench improves average performance up to 25.8% over closed-source
  GPT-4o in document reasoning and structured output tasks such as
  Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a
  preference for outputs from models trained on BigDocs over GPT-4o. This
  suggests that BigDocs can help both academics and the open-source community
  utilize and improve AI tools to enhance multimodal capabilities and document
  reasoning. The project is hosted at https://bigdocs.github.io .