title: Lumos : Empowering Multimodal LLMs with Scene Text Recognition
summary: We introduce Lumos, the first end-to-end multimodal question-answering system
  with text understanding capabilities. At the core of Lumos is a Scene Text
  Recognition (STR) component that extracts text from first person point-of-view
  images, the output of which is used to augment input to a Multimodal Large
  Language Model (MM-LLM). While building Lumos, we encountered numerous
  challenges related to STR quality, overall latency, and model inference. In
  this paper, we delve into those challenges, and discuss the system
  architecture, design choices, and modeling techniques employed to overcome
  these obstacles. We also provide a comprehensive evaluation for each component,
  showcasing high quality and efficiency.
paper_authors._id: 65cc2ce019683f9817234084
paper_authors.name: Ashish Shenoy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-15T08:28:04.992Z
paper_authors._id: 65cce8307b7a844001bf9fff
paper_authors.avatarUrl: /avatars/130d7736cfdd00af1ba9897b79ee7c82.svg
paper_authors.fullname: Ashish Shenoy
paper_authors.type: user
paper_authors.user: ashishvshenoy
paper_authors._id: 65cc2ce019683f9817234085
paper_authors.name: Yichao Lu
paper_authors._id: 65cc2ce019683f9817234086
paper_authors.name: Srihari Jayakumar
paper_authors._id: 65cc2ce019683f9817234087
paper_authors.name: Debojeet Chatterjee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T12:55:55.009Z
paper_authors._id: 6261c8376c57f74478073c3a
paper_authors.avatarUrl: /avatars/8a64178dd148379b298b0bced79295e8.svg
paper_authors.fullname: Debojeet Chatterjee
paper_authors.type: user
paper_authors.user: debowin
paper_authors._id: 65cc2ce019683f9817234088
paper_authors.name: Mohsen Moslehpour
paper_authors._id: 65cc2ce019683f9817234089
paper_authors.name: Pierce Chuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T12:56:04.997Z
paper_authors._id: 640bb0a619f883e24a4a0f1e
paper_authors.avatarUrl: /avatars/c884d5ed8009dffb3f9414471a8817e3.svg
paper_authors.fullname: Pierce Chuang
paper_authors.type: user
paper_authors.user: pichuang
paper_authors._id: 65cc2ce019683f981723408a
paper_authors.name: Abhay Harpale
paper_authors._id: 65cc2ce019683f981723408b
paper_authors.name: Vikas Bhardwaj
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-14T16:06:55.944Z
paper_authors._id: 628830d687a393458fc33a1c
paper_authors.avatarUrl: /avatars/fc8786ceef9188efd19f7ae5b9e30554.svg
paper_authors.fullname: Vikas Bhardwaj
paper_authors.type: user
paper_authors.user: vikasbhardwaj
paper_authors._id: 65cc2ce019683f981723408c
paper_authors.name: Di Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T12:56:28.096Z
paper_authors._id: 638194c94761ddfa0091a981
paper_authors.avatarUrl: /avatars/ca9974ac8f3e4bdd2806f65e64b383f8.svg
paper_authors.fullname: Di Xu
paper_authors.type: user
paper_authors.user: dixu
paper_authors._id: 65cc2ce019683f981723408d
paper_authors.name: Shicong Zhao
paper_authors._id: 65cc2ce019683f981723408e
paper_authors.name: Longfang Zhao
paper_authors._id: 65cc2ce019683f981723408f
paper_authors.name: Ankit Ramchandani
paper_authors._id: 65cc2ce019683f9817234090
paper_authors.name: Xin Luna Dong
paper_authors._id: 65cc2ce019683f9817234091
paper_authors.name: Anuj Kumar
paper_title: Lumos : Empowering Multimodal LLMs with Scene Text Recognition
paper_summary: We introduce Lumos, the first end-to-end multimodal question-answering system
  with text understanding capabilities. At the core of Lumos is a Scene Text
  Recognition (STR) component that extracts text from first person point-of-view
  images, the output of which is used to augment input to a Multimodal Large
  Language Model (MM-LLM). While building Lumos, we encountered numerous
  challenges related to STR quality, overall latency, and model inference. In
  this paper, we delve into those challenges, and discuss the system
  architecture, design choices, and modeling techniques employed to overcome
  these obstacles. We also provide a comprehensive evaluation for each component,
  showcasing high quality and efficiency.
paper_ai_keywords: Scene Text Recognition (STR) Multimodal Large Language Model (MM-LLM) first person point-of-view images end-to-end multimodal question-answering system system architecture design choices modeling techniques