title: OmniFusion Technical Report
summary: Last year, multimodal architectures served up a revolution in AI-based
  approaches and solutions, extending the capabilities of large language models
  (LLM). We propose an OmniFusion model based on a pretrained LLM and
  adapters for visual modality. We evaluated and compared several architecture
  design principles for better text and visual data coupling: MLP and transformer
  adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their
  fusing approach, image encoding method (whole image or tiles encoding) and two
  7B LLMs (the proprietary one and open-source Mistral). Experiments on 8
  visual-language benchmarks show the top score for the best OmniFusion setup in
  terms of different VQA tasks in comparison with open-source LLaVA-like
  solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We
  also propose a variety of situations, where OmniFusion provides highly-detailed
  answers in different domains: housekeeping, sightseeing, culture, medicine,
  handwritten and scanned equations recognition, etc. Mistral-based OmniFusion
  model is an open-source solution with weights, training and inference scripts
  available at https://github.com/AIRI-Institute/OmniFusion.
paper_authors._id: 6615f762c7f45b3f8930674d
paper_authors.name: Elizaveta Goncharova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:37:04.576Z
paper_authors._id: 6310ff34bc152fa3e810c186
paper_authors.avatarUrl: /avatars/bfd63bcd81548283f5e496e3693bf143.svg
paper_authors.fullname: Elizaveta Goncharova
paper_authors.type: user
paper_authors.user: Elizaveta
paper_authors._id: 6615f762c7f45b3f8930674e
paper_authors.name: Anton Razzhigaev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-10T10:58:25.360Z
paper_authors._id: 6172aaeec8e66e2aa84c06b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6172aaeec8e66e2aa84c06b9/ZdRZSp3P1SU6CIDbvQwkv.jpeg
paper_authors.fullname: Anton Razzhigaev
paper_authors.type: user
paper_authors.user: razzant
paper_authors._id: 6615f762c7f45b3f8930674f
paper_authors.name: Matvey Mikhalchuk
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T05:52:36.743Z
paper_authors._id: 64ee45a944f4b3b1bccc02d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ee45a944f4b3b1bccc02d1/SoidO9HQ4mftzbUPtuBBf.png
paper_authors.fullname: Matvey Mikhalchuk
paper_authors.type: user
paper_authors.user: matveymih
paper_authors._id: 6615f762c7f45b3f89306750
paper_authors.name: Maxim Kurkin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-10T10:58:16.683Z
paper_authors._id: 634c084c4abe84057588fd63
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634c084c4abe84057588fd63/-3doVHYwCqbkPQtwVX9aM.jpeg
paper_authors.fullname: Maxim Kurkin
paper_authors.type: user
paper_authors.user: dondosss
paper_authors._id: 6615f762c7f45b3f89306751
paper_authors.name: Irina Abdullaeva
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T14:54:46.017Z
paper_authors._id: 648444f14bb88d273c4cab7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648444f14bb88d273c4cab7f/iGjkOf34bcB6K2-J5C_0U.jpeg
paper_authors.fullname: Irina Abdullaeva
paper_authors.type: user
paper_authors.user: IrinaAbdullaeva
paper_authors._id: 6615f762c7f45b3f89306752
paper_authors.name: Matvey Skripkin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-23T07:15:41.309Z
paper_authors._id: 6626c5d0a329de26e7eb16fa
paper_authors.avatarUrl: /avatars/124f389f768fb666efd8b5a9b54c3b3c.svg
paper_authors.fullname: Matvey Skripkin
paper_authors.type: user
paper_authors.user: barracuda049
paper_authors._id: 6615f762c7f45b3f89306753
paper_authors.name: Ivan Oseledets
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:37:10.732Z
paper_authors._id: 6169a581d05945bfd8718dfa
paper_authors.avatarUrl: /avatars/1892ab06a7ddb557232777de3cbec470.svg
paper_authors.fullname: Ivan Oseledets
paper_authors.type: user
paper_authors.user: oseledets
paper_authors._id: 6615f762c7f45b3f89306754
paper_authors.name: Denis Dimitrov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T13:07:37.859Z
paper_authors._id: 6669a678465d1d802181e456
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6669a678465d1d802181e456/ZCthBBhDFQnh0bBkgUQUU.png
paper_authors.fullname: Denis Dimitrov
paper_authors.type: user
paper_authors.user: dendimitrov
paper_authors._id: 6615f762c7f45b3f89306755
paper_authors.name: Andrey Kuznetsov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T05:52:42.541Z
paper_authors._id: 643984dceb7c5616ef3f5d54
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
paper_authors.fullname: Andrey Kuznetsov
paper_authors.type: user
paper_authors.user: kuznetsoffandrey
paper_title: OmniFusion Technical Report
paper_summary: Last year, multimodal architectures served up a revolution in AI-based
  approaches and solutions, extending the capabilities of large language models
  (LLM). We propose an OmniFusion model based on a pretrained LLM and
  adapters for visual modality. We evaluated and compared several architecture
  design principles for better text and visual data coupling: MLP and transformer
  adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their
  fusing approach, image encoding method (whole image or tiles encoding) and two
  7B LLMs (the proprietary one and open-source Mistral). Experiments on 8
  visual-language benchmarks show the top score for the best OmniFusion setup in
  terms of different VQA tasks in comparison with open-source LLaVA-like
  solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We
  also propose a variety of situations, where OmniFusion provides highly-detailed
  answers in different domains: housekeeping, sightseeing, culture, medicine,
  handwritten and scanned equations recognition, etc. Mistral-based OmniFusion
  model is an open-source solution with weights, training and inference scripts
  available at https://github.com/AIRI-Institute/OmniFusion.
paper_ai_keywords: OmniFusion pretrained LLM adapters visual modality MLP and transformer adapters CLIP ViT-based encoders SigLIP InternVIT image encoding whole image tiles encoding VQA tasks VizWiz Pope MM-Vet ScienceQA MMBench TextVQA VQAv2 MMMU Mistral-based OmniFusion