title: GLOV: Guided Large Language Models as Implicit Optimizers for Vision
    Language Models
summary: In this work, we propose a novel method (GLOV) enabling Large Language Models
  (LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to
  enhance downstream vision tasks. Our GLOV meta-prompts an LLM with the
  downstream task description, querying it for suitable VLM prompts (e.g., for
  zero-shot classification with CLIP). These prompts are ranked according to a
  purity measure obtained through a fitness function. In each respective
  optimization step, the ranked prompts are fed as in-context examples (with
  their accuracies) to equip the LLM with the knowledge of the type of text
  prompts preferred by the downstream VLM. Furthermore, we also explicitly steer
  the LLM generation process in each optimization step by specifically adding an
  offset difference vector of the embeddings from the positive and negative
  solutions found by the LLM, in previous optimization steps, to the intermediate
  layer of the network for the next generation step. This offset vector steers
  the LLM generation toward the type of language preferred by the downstream VLM,
  resulting in enhanced performance on the downstream vision tasks. We
  comprehensively evaluate our GLOV on 16 diverse datasets using two families of
  VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models
  -- showing that the discovered solutions can enhance the recognition
  performance by up to 15.0% and 57.5% (3.8% and 21.6% on average) for these
  models.
paper_authors._id: 6708cbe58970d752b72c4d6e
paper_authors.name: M. Jehanzeb Mirza
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T13:07:28.583Z
paper_authors._id: 64ce40fee13e7c2bc5865bf6
paper_authors.avatarUrl: /avatars/2b9f377ebe23d703a7941c54c3aacd98.svg
paper_authors.fullname: Muhammad Jehanzeb Mirza
paper_authors.type: user
paper_authors.user: jmiemirza
paper_authors._id: 6708cbe58970d752b72c4d6f
paper_authors.name: Mengjie Zhao
paper_authors._id: 6708cbe58970d752b72c4d70
paper_authors.name: Zhuoyuan Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:23:03.689Z
paper_authors._id: 64365f641c4caf9cc9e1a95a
paper_authors.avatarUrl: /avatars/7be333e4f80e668f951d1e4c7351b89b.svg
paper_authors.fullname: Zhuoyuan Mao
paper_authors.type: user
paper_authors.user: kevinmzy
paper_authors._id: 6708cbe58970d752b72c4d71
paper_authors.name: Sivan Doveh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:22:26.292Z
paper_authors._id: 6255ad878e2cfccdb435f96c
paper_authors.avatarUrl: /avatars/13c6e9ed14ef1de2875645b8db5da006.svg
paper_authors.fullname: sivan doveh
paper_authors.type: user
paper_authors.user: sivand
paper_authors._id: 6708cbe58970d752b72c4d72
paper_authors.name: Wei Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:24.074Z
paper_authors._id: 64b99a3fd4463c3d2183739d
paper_authors.avatarUrl: /avatars/9b0fddbf6b13dd1baa7cfcee3ff4f0b9.svg
paper_authors.fullname: Wei Lin
paper_authors.type: user
paper_authors.user: wlin21at
paper_authors._id: 6708cbe58970d752b72c4d73
paper_authors.name: Paul Gavrikov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:22:19.804Z
paper_authors._id: 6266c07e7a1f5a1562c4113b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6266c07e7a1f5a1562c4113b/GEt70X7-UmNIbECDA0eVs.jpeg
paper_authors.fullname: Paul Gavrikov
paper_authors.type: user
paper_authors.user: paulgavrikov
paper_authors._id: 6708cbe58970d752b72c4d74
paper_authors.name: Michael Dorkenwald
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:22:13.730Z
paper_authors._id: 6566436e3d526e3297c1b4a1
paper_authors.avatarUrl: /avatars/e433ffccb0f9c2d91649f9988655fedd.svg
paper_authors.fullname: Michael Dorkenwald
paper_authors.type: user
paper_authors.user: mdorkenw
paper_authors._id: 6708cbe58970d752b72c4d75
paper_authors.name: Shiqi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T11:23:23.156Z
paper_authors._id: 6432a3bcfc29acb96be55640
paper_authors.avatarUrl: /avatars/a42670a8008ade62edc7f0fe5af4bbaf.svg
paper_authors.fullname: Shiqi
paper_authors.type: user
paper_authors.user: aquila147
paper_authors._id: 6708cbe58970d752b72c4d76
paper_authors.name: Saurav Jha
paper_authors._id: 6708cbe58970d752b72c4d77
paper_authors.name: Hiromi Wakaki
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:22:00.047Z
paper_authors._id: 665e7c9b045fcbf12b961e78
paper_authors.avatarUrl: /avatars/dfb7d19dc0104b0520840a5b3984e998.svg
paper_authors.fullname: Hiromi Wakaki
paper_authors.type: user
paper_authors.user: HiromiWakaki
paper_authors._id: 6708cbe58970d752b72c4d78
paper_authors.name: Yuki Mitsufuji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:21:54.311Z
paper_authors._id: 665e32384ecc8a7181634f6d
paper_authors.avatarUrl: /avatars/8752f952010540d14f45eac849e91371.svg
paper_authors.fullname: Yuki Mitsufuji
paper_authors.type: user
paper_authors.user: mittu1204
paper_authors._id: 6708cbe58970d752b72c4d79
paper_authors.name: Horst Possegger
paper_authors._id: 6708cbe58970d752b72c4d7a
paper_authors.name: Rogerio Feris
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:21:43.251Z
paper_authors._id: 6426053ca5ec4a5cbc516567
paper_authors.avatarUrl: /avatars/7b841e0c58aa746e662fc8c78e3c5537.svg
paper_authors.fullname: Rogerio Feris
paper_authors.type: user
paper_authors.user: rferis
paper_authors._id: 6708cbe58970d752b72c4d7b
paper_authors.name: Leonid Karlinsky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:21:37.830Z
paper_authors._id: 62558f7cebfc059817b06690
paper_authors.avatarUrl: /avatars/40fbfdbeb0b33e21a48478deb2a742d1.svg
paper_authors.fullname: Leonid Karlinsky
paper_authors.type: user
paper_authors.user: leokarlin
paper_authors._id: 6708cbe58970d752b72c4d7c
paper_authors.name: James Glass
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T11:21:30.774Z
paper_authors._id: 62a793d33e101ec156cc58c4
paper_authors.avatarUrl: /avatars/d3607cdb2cac1b2a3011d7441e6ea321.svg
paper_authors.fullname: James Glass
paper_authors.type: user
paper_authors.user: OmegaLittleBob
paper_title: GLOV: Guided Large Language Models as Implicit Optimizers for Vision
    Language Models
paper_summary: In this work, we propose a novel method (GLOV) enabling Large Language Models
  (LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to
  enhance downstream vision tasks. Our GLOV meta-prompts an LLM with the
  downstream task description, querying it for suitable VLM prompts (e.g., for
  zero-shot classification with CLIP). These prompts are ranked according to a
  purity measure obtained through a fitness function. In each respective
  optimization step, the ranked prompts are fed as in-context examples (with
  their accuracies) to equip the LLM with the knowledge of the type of text
  prompts preferred by the downstream VLM. Furthermore, we also explicitly steer
  the LLM generation process in each optimization step by specifically adding an
  offset difference vector of the embeddings from the positive and negative
  solutions found by the LLM, in previous optimization steps, to the intermediate
  layer of the network for the next generation step. This offset vector steers
  the LLM generation toward the type of language preferred by the downstream VLM,
  resulting in enhanced performance on the downstream vision tasks. We
  comprehensively evaluate our GLOV on 16 diverse datasets using two families of
  VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models
  -- showing that the discovered solutions can enhance the recognition
  performance by up to 15.0% and 57.5% (3.8% and 21.6% on average) for these
  models.
paper_ai_summary: GLOV method enhances vision tasks by leveraging LLMs as implicit optimizers for VLMs through meta-prompts and offset vector guidance.
paper_ai_keywords: Large Language Models LLMs Vision-Language Models VLMs GLOV meta-prompts zero-shot classification CLIP fitness function in-context examples offset difference vector dual-encoder encoder-decoder LLaVa recognition performance