title: MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
summary: With advances in foundational and vision-language models, and effective
  fine-tuning techniques, a large number of both general and special-purpose
  models have been developed for a variety of visual tasks. Despite the
  flexibility and accessibility of these models, no single model is able to
  handle all tasks and/or applications that may be envisioned by potential users.
  Recent approaches, such as visual programming and multimodal LLMs with
  integrated tools aim to tackle complex visual tasks, by way of program
  synthesis. However, such approaches overlook user constraints (e.g.,
  performance / computational needs), produce test-time sample-specific solutions
  that are difficult to deploy, and, sometimes, require low-level instructions
  that maybe beyond the abilities of a naive user. To address these limitations,
  we introduce MMFactory, a universal framework that includes model and metrics
  routing components, acting like a solution search engine across various
  available models. Based on a task description and few sample input-output pairs
  and (optionally) resource and/or performance constraints, MMFactory can suggest
  a diverse pool of programmatic solutions by instantiating and combining
  visio-lingual tools from its model repository. In addition to synthesizing
  these solutions, MMFactory also proposes metrics and benchmarks performance /
  resource characteristics, allowing users to pick a solution that meets their
  unique design constraints. From the technical perspective, we also introduced a
  committee-based solution proposer that leverages multi-agent LLM conversation
  to generate executable, diverse, universal, and robust solutions for the user.
  Experimental results show that MMFactory outperforms existing methods by
  delivering state-of-the-art solutions tailored to user problem specifications.
  Project page is available at https://davidhalladay.github.io/mmfactory_demo.
paper_authors._id: 676e9cfc11998b72ab00be60
paper_authors.name: Wan-Cyuan Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:13:13.288Z
paper_authors._id: 63357efea01bd734f7243c61
paper_authors.avatarUrl: /avatars/459d3e35bb6ae60f69a0c7bd77a39d58.svg
paper_authors.fullname: WanCyuan Fan
paper_authors.type: user
paper_authors.user: ChrisFan
paper_authors._id: 676e9cfc11998b72ab00be61
paper_authors.name: Tanzila Rahman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:12:47.649Z
paper_authors._id: 64062c3ca577649430c1006b
paper_authors.avatarUrl: /avatars/870ac9e8dc15051e97a7a6efd46e8a36.svg
paper_authors.fullname: Tanzila Rahman
paper_authors.type: user
paper_authors.user: trahman
paper_authors._id: 676e9cfc11998b72ab00be62
paper_authors.name: Leonid Sigal
paper_title: MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
paper_summary: With advances in foundational and vision-language models, and effective
  fine-tuning techniques, a large number of both general and special-purpose
  models have been developed for a variety of visual tasks. Despite the
  flexibility and accessibility of these models, no single model is able to
  handle all tasks and/or applications that may be envisioned by potential users.
  Recent approaches, such as visual programming and multimodal LLMs with
  integrated tools aim to tackle complex visual tasks, by way of program
  synthesis. However, such approaches overlook user constraints (e.g.,
  performance / computational needs), produce test-time sample-specific solutions
  that are difficult to deploy, and, sometimes, require low-level instructions
  that maybe beyond the abilities of a naive user. To address these limitations,
  we introduce MMFactory, a universal framework that includes model and metrics
  routing components, acting like a solution search engine across various
  available models. Based on a task description and few sample input-output pairs
  and (optionally) resource and/or performance constraints, MMFactory can suggest
  a diverse pool of programmatic solutions by instantiating and combining
  visio-lingual tools from its model repository. In addition to synthesizing
  these solutions, MMFactory also proposes metrics and benchmarks performance /
  resource characteristics, allowing users to pick a solution that meets their
  unique design constraints. From the technical perspective, we also introduced a
  committee-based solution proposer that leverages multi-agent LLM conversation
  to generate executable, diverse, universal, and robust solutions for the user.
  Experimental results show that MMFactory outperforms existing methods by
  delivering state-of-the-art solutions tailored to user problem specifications.
  Project page is available at https://davidhalladay.github.io/mmfactory_demo.
paper_ai_keywords: compos-language models fine-tuning techniques foundational models vision-language models visual programming multimodal LLMs program synthesis sample-specific solutions MMFactory model and metrics routing solution search engine task description input-output pairs programmatic solutions committee-based solution-proposer multi-agent LLM conversation executable solutions diverse solutions robust solutions state-of-the-Art Solutions