🧑‍🍳 New Multimodal Fine-Tuning Recipe 🧑‍🍳
text
🧑‍🍳 New Multimodal Fine-Tuning Recipe 🧑‍🍳
new_line
new_line
⚡️ In this new 
text
⚡️ In this new 
@huggingface
mention
huggingface
 Cookbook recipe, I walk you though the process of fine tuning a Visual Language Model (VLM) for Object Detection with Visual Grounding, using TRL.
text
 Cookbook recipe, I walk you though the process of fine tuning a Visual Language Model (VLM) for Object Detection with Visual Grounding, using TRL.
new_line
new_line
🔍 Object detection typically involves detecting categories in images (e.g., vase). 
text
🔍 Object detection typically involves detecting categories in images (e.g., vase). 
new_line
new_line
By combining it with visual grounding, we add contextual understanding so instead of detecting just "vase", we can detect "middle vase" in an image. 
text
By combining it with visual grounding, we add contextual understanding so instead of detecting just "vase", we can detect "middle vase" in an image. 
new_line
new_line
VLMs are super powerful!
text
VLMs are super powerful!
new_line
new_line
In this case, I use PaliGemma 2 which already supports object detection and extend it to also add visual grounding.
text
In this case, I use PaliGemma 2 which already supports object detection and extend it to also add visual grounding.
new_line
new_line
🤗 Check it out here: 
text
🤗 Check it out here: 
https://huggingface.co/learn/cookbook/fine_tuning_vlm_object_detection_grounding
https://huggingface.co/learn/cookbook/fine_tuning_vlm_object_detection_grounding
link
🧑‍🍳 New Multimodal Fine-Tuning Recipe 🧑‍🍳

⚡️ In this new @huggingface Cookbook recipe, I walk you though the process of fine tuning a Visual Language Model (VLM) for Object Detection with Visual Grounding, using TRL.

🔍 Object detection typically involves detecting categories in images (e.g., vase). 

By combining it with visual grounding, we add contextual understanding so instead of detecting just "vase", we can detect "middle vase" in an image. 

VLMs are super powerful!

In this case, I use PaliGemma 2 which already supports object detection and extend it to also add visual grounding.

🤗 Check it out here: https://huggingface.co/learn/cookbook/fine_tuning_vlm_object_detection_grounding
👍
nqzfaizal77ai John6666 Dcas89 miseyu