GLOV: Guided Large Language Models as Implicit Optimizers for Vision
  Language Models
In this work, we propose a novel method (GLOV) enabling Large Language Models
(LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to
enhance downstream vision tasks. Our GLOV meta-prompts an LLM with the
downstream task description, querying it for suitable VLM prompts (e.g., for
zero-shot classification with CLIP). These prompts are ranked according to a
purity measure obtained through a fitness function. In each respective
optimization step, the ranked prompts are fed as in-context examples (with
their accuracies) to equip the LLM with the knowledge of the type of text
prompts preferred by the downstream VLM. Furthermore, we also explicitly steer
the LLM generation process in each optimization step by specifically adding an
offset difference vector of the embeddings from the positive and negative
solutions found by the LLM, in previous optimization steps, to the intermediate
layer of the network for the next generation step. This offset vector steers
the LLM generation toward the type of language preferred by the downstream VLM,
resulting in enhanced performance on the downstream vision tasks. We
comprehensively evaluate our GLOV on 16 diverse datasets using two families of
VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models
-- showing that the discovered solutions can enhance the recognition
performance by up to 15.0% and 57.5% (3.8% and 21.6% on average) for these
models.
6708cbe58970d752b72c4d6e
M. Jehanzeb Mirza
claimed_verified
2024-10-11T13:07:28.583Z
64ce40fee13e7c2bc5865bf6
/avatars/2b9f377ebe23d703a7941c54c3aacd98.svg
Muhammad Jehanzeb Mirza
user
jmiemirza
6708cbe58970d752b72c4d6f
Mengjie Zhao
6708cbe58970d752b72c4d70
Zhuoyuan Mao
admin_assigned
2024-10-11T11:23:03.689Z
64365f641c4caf9cc9e1a95a
/avatars/7be333e4f80e668f951d1e4c7351b89b.svg
Zhuoyuan Mao
user
kevinmzy
6708cbe58970d752b72c4d71
Sivan Doveh
admin_assigned
2024-10-11T11:22:26.292Z
6255ad878e2cfccdb435f96c
/avatars/13c6e9ed14ef1de2875645b8db5da006.svg
sivan doveh
user
sivand
6708cbe58970d752b72c4d72
Wei Lin
claimed_verified
2024-10-15T09:11:24.074Z
64b99a3fd4463c3d2183739d
/avatars/9b0fddbf6b13dd1baa7cfcee3ff4f0b9.svg
Wei Lin
user
wlin21at
6708cbe58970d752b72c4d73
Paul Gavrikov
admin_assigned
2024-10-11T11:22:19.804Z
6266c07e7a1f5a1562c4113b
https://cdn-avatars.huggingface.co/v1/production/uploads/6266c07e7a1f5a1562c4113b/GEt70X7-UmNIbECDA0eVs.jpeg
Paul Gavrikov
user
paulgavrikov
6708cbe58970d752b72c4d74
Michael Dorkenwald
admin_assigned
2024-10-11T11:22:13.730Z
6566436e3d526e3297c1b4a1
/avatars/e433ffccb0f9c2d91649f9988655fedd.svg
Michael Dorkenwald
user
mdorkenw
6708cbe58970d752b72c4d75
Shiqi Yang
claimed_verified
2024-10-11T11:23:23.156Z
6432a3bcfc29acb96be55640
/avatars/a42670a8008ade62edc7f0fe5af4bbaf.svg
Shiqi
user
aquila147
6708cbe58970d752b72c4d76
Saurav Jha
6708cbe58970d752b72c4d77
Hiromi Wakaki
admin_assigned
2024-10-11T11:22:00.047Z
665e7c9b045fcbf12b961e78
/avatars/dfb7d19dc0104b0520840a5b3984e998.svg
Hiromi Wakaki
user
HiromiWakaki
6708cbe58970d752b72c4d78
Yuki Mitsufuji
admin_assigned
2024-10-11T11:21:54.311Z
665e32384ecc8a7181634f6d
/avatars/8752f952010540d14f45eac849e91371.svg
Yuki Mitsufuji
user
mittu1204
6708cbe58970d752b72c4d79
Horst Possegger
6708cbe58970d752b72c4d7a
Rogerio Feris
admin_assigned
2024-10-11T11:21:43.251Z
6426053ca5ec4a5cbc516567
/avatars/7b841e0c58aa746e662fc8c78e3c5537.svg
Rogerio Feris
user
rferis
6708cbe58970d752b72c4d7b
Leonid Karlinsky
admin_assigned
2024-10-11T11:21:37.830Z
62558f7cebfc059817b06690
/avatars/40fbfdbeb0b33e21a48478deb2a742d1.svg
Leonid Karlinsky
user
leokarlin
6708cbe58970d752b72c4d7c
James Glass
admin_assigned
2024-10-11T11:21:30.774Z
62a793d33e101ec156cc58c4
/avatars/d3607cdb2cac1b2a3011d7441e6ea321.svg
James Glass
user
OmegaLittleBob
GLOV: Guided Large Language Models as Implicit Optimizers for Vision
  Language Models
In this work, we propose a novel method (GLOV) enabling Large Language Models
(LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to
enhance downstream vision tasks. Our GLOV meta-prompts an LLM with the
downstream task description, querying it for suitable VLM prompts (e.g., for
zero-shot classification with CLIP). These prompts are ranked according to a
purity measure obtained through a fitness function. In each respective
optimization step, the ranked prompts are fed as in-context examples (with
their accuracies) to equip the LLM with the knowledge of the type of text
prompts preferred by the downstream VLM. Furthermore, we also explicitly steer
the LLM generation process in each optimization step by specifically adding an
offset difference vector of the embeddings from the positive and negative
solutions found by the LLM, in previous optimization steps, to the intermediate
layer of the network for the next generation step. This offset vector steers
the LLM generation toward the type of language preferred by the downstream VLM,
resulting in enhanced performance on the downstream vision tasks. We
comprehensively evaluate our GLOV on 16 diverse datasets using two families of
VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models
-- showing that the discovered solutions can enhance the recognition
performance by up to 15.0% and 57.5% (3.8% and 21.6% on average) for these
models.
GLOV method enhances vision tasks by leveraging LLMs as implicit optimizers for VLMs through meta-prompts and offset vector guidance.
Large Language Models LLMs Vision-Language Models VLMs GLOV meta-prompts zero-shot classification CLIP fitness function in-context examples offset difference vector dual-encoder encoder-decoder LLaVa recognition performance