content.type: new_line
content.raw: After the Supervised Fine-Tuning (SFT) phase, we observed a notable degradation in the instruction-following capabilities of the LLaVA Multi-Modal Large Language Model (MM-LLM). To address this issue, we introduced a 6K-entry VQA preference dataset and employed Direct Preference Optimization (DPO), alongside testing other algorithms such as Rejection Sampling and SteerLM, to enhance instruction-following proficiency. Our methodology not only fully restored the language following capabilities of LLaVa on the MT-Bench but also outperformed LLaVA-RLHF and Vicuna. Additionally, our approach extended to visual VQA tasks, as demonstrated by significant performance improvements on MM-Vet and LLaVa-Bench. An interesting observation was that, compared to models using distilled SFT, our method showed substantial out-of-distribution improvements. 
content.type: text
content.value: After the Supervised Fine-Tuning (SFT) phase, we observed a notable degradation in the instruction-following capabilities of the LLaVA Multi-Modal Large Language Model (MM-LLM). To address this issue, we introduced a 6K-entry VQA preference dataset and employed Direct Preference Optimization (DPO), alongside testing other algorithms such as Rejection Sampling and SteerLM, to enhance instruction-following proficiency. Our methodology not only fully restored the language following capabilities of LLaVa on the MT-Bench but also outperformed LLaVA-RLHF and Vicuna. Additionally, our approach extended to visual VQA tasks, as demonstrated by significant performance improvements on MM-Vet and LLaVa-Bench. An interesting observation was that, compared to models using distilled SFT, our method showed substantial out-of-distribution improvements. 
content.type: new_line
content.type: new_line
content.href: https://arxiv.org/abs/2402.10884
content.raw: https://arxiv.org/abs/2402.10884
content.type: link
content.type: new_line
content.raw: Model available
content.type: text
content.value: Model available
content.type: new_line
content.raw: https://huggingface.co/alexshengzhili/llava-v1.5-13b-dpo
content.id: alexshengzhili/llava-v1.5-13b-dpo
content.type: model
content.type: resource
content.url: https://huggingface.co/alexshengzhili/llava-v1.5-13b-dpo
content.type: new_line
content.raw: GitHub: 
content.type: text
content.value: GitHub: 
content.type: new_line
content.href: https://github.com/findalexli/mllm-dpo/edit/main/README.MD
content.raw: https://github.com/findalexli/mllm-dpo/edit/main/README.MD
content.type: link
rawContent: 
  After the Supervised Fine-Tuning (SFT) phase, we observed a notable degradation in the instruction-following capabilities of the LLaVA Multi-Modal Large Language Model (MM-LLM). To address this issue, we introduced a 6K-entry VQA preference dataset and employed Direct Preference Optimization (DPO), alongside testing other algorithms such as Rejection Sampling and SteerLM, to enhance instruction-following proficiency. Our methodology not only fully restored the language following capabilities of LLaVa on the MT-Bench but also outperformed LLaVA-RLHF and Vicuna. Additionally, our approach extended to visual VQA tasks, as demonstrated by significant performance improvements on MM-Vet and LLaVa-Bench. An interesting observation was that, compared to models using distilled SFT, our method showed substantial out-of-distribution improvements. 
  
  https://arxiv.org/abs/2402.10884
  Model available
  https://huggingface.co/alexshengzhili/llava-v1.5-13b-dpo
  GitHub: 
  https://github.com/findalexli/mllm-dpo/edit/main/README.MD
reactions.reaction: ❤️
reactions.users: samusenps osanseviero cstr