Meta Reality Labs has developed Lumos, a system that merges Multimodal Large Language Models (MM-LLMs) with Scene Text Recognition (STR) to boost the efficiency of various tasks such as multimodal question-answering and text summarization.
text
Meta Reality Labs has developed Lumos, a system that merges Multimodal Large Language Models (MM-LLMs) with Scene Text Recognition (STR) to boost the efficiency of various tasks such as multimodal question-answering and text summarization.
new_line
new_line
Key aspects of Lumos include:
text
Key aspects of Lumos include:
new_line
new_line
* Hybrid Computing: Utilizes a combination of on-device and cloud computing to process inputs, aiming to reduce latency.
text
* Hybrid Computing: Utilizes a combination of on-device and cloud computing to process inputs, aiming to reduce latency.
new_line
* STR Components:
text
* STR Components:
new_line
  * Region of Interest (ROI) Detection: Focuses on text-rich areas within images for optimized text extraction.
text
  * Region of Interest (ROI) Detection: Focuses on text-rich areas within images for optimized text extraction.
new_line
  * Text Detection and Recognition: Ensures high-quality text recognition within the ROI.
text
  * Text Detection and Recognition: Ensures high-quality text recognition within the ROI.
new_line
  * Reading Order Reconstruction: Arranges recognized text to mimic natural reading order, essential for context understanding.
text
  * Reading Order Reconstruction: Arranges recognized text to mimic natural reading order, essential for context understanding.
new_line
new_line
Lumos demonstrates significant improvement with 80% accuracy in question-answering benchmarks and a low word error rate.
text
Lumos demonstrates significant improvement with 80% accuracy in question-answering benchmarks and a low word error rate.
new_line
new_line
Paper: 
text
Paper: 
Lumos : Empowering Multimodal LLMs with Scene Text Recognition (2402.08017)
https://huggingface.co/papers/2402.08017
2402.08017
paper
resource
https://huggingface.co/papers/2402.08017
new_line
new_line
Congrats to the authors for their work!
text
Congrats to the authors for their work!
Meta Reality Labs has developed Lumos, a system that merges Multimodal Large Language Models (MM-LLMs) with Scene Text Recognition (STR) to boost the efficiency of various tasks such as multimodal question-answering and text summarization.

Key aspects of Lumos include:

* Hybrid Computing: Utilizes a combination of on-device and cloud computing to process inputs, aiming to reduce latency.
* STR Components:
  * Region of Interest (ROI) Detection: Focuses on text-rich areas within images for optimized text extraction.
  * Text Detection and Recognition: Ensures high-quality text recognition within the ROI.
  * Reading Order Reconstruction: Arranges recognized text to mimic natural reading order, essential for context understanding.

Lumos demonstrates significant improvement with 80% accuracy in question-answering benchmarks and a low word error rate.

Paper: https://huggingface.co/papers/2402.08017

Congrats to the authors for their work!
👍
Dlbk ybelkada ajibawa-2023 musfiqdehan osanseviero clem Esquare jjoker999 Mrdaffy bunnycore QuantumResearch merve ioanacroi
🤯
alielfilali01 Dlbk jeffboudier merve
❤️
alielfilali01 Dlbk clem merve
🤝
deepufrk Dlbk merve
6580e99d8108945fd3b88570
/avatars/8658ce5c1ad100593b3679c8c97e6f42.svg
Quantum Research
QuantumResearch
user
657217faabb25ed8aedd5e48
https://cdn-avatars.huggingface.co/v1/production/uploads/657217faabb25ed8aedd5e48/UUHAXeGtOnQBXFD3nYtf2.jpeg
Vlad Bogolin
vladbogo
user