title: Florenz: Scaling Laws for Systematic Generalization in Vision-Language
    Models
summary: Cross-lingual transfer enables vision-language models (VLMs) to perform
  vision tasks in various languages with training data only in one language.
  Current approaches rely on large pre-trained multilingual language models.
  However, they face the curse of multilinguality, sacrificing downstream task
  performance for multilingual capabilities, struggling with lexical ambiguities,
  and falling behind recent advances. In this work, we study the scaling laws of
  systematic generalization with monolingual VLMs for multilingual tasks,
  focusing on the impact of model size and seen training samples. We propose
  Florenz, a monolingual encoder-decoder VLM with 0.4B to 11.2B parameters
  combining the pre-trained VLM Florence-2 and the large language model Gemma-2.
  Florenz is trained with varying compute budgets on a synthetic dataset that
  features intentionally incomplete language coverage for image captioning, thus,
  testing generalization from the fully covered translation task. We show that
  not only does indirectly learning unseen task-language pairs adhere to a
  scaling law, but also that with our data generation pipeline and the proposed
  Florenz model family, image captioning abilities can emerge in a specific
  language even when only data for the translation task is available. Fine-tuning
  on a mix of downstream datasets yields competitive performance and demonstrates
  promising scaling trends in multimodal machine translation (Multi30K, CoMMuTE),
  lexical disambiguation (CoMMuTE), and image captioning (Multi30K, XM3600, COCO
  Karpathy).
paper_authors._id: 67da8b794e1138ddc328de09
paper_authors.name: Julian Spravil
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:43:48.893Z
paper_authors._id: 630a4aaa9df54451d91cc6fa
paper_authors.avatarUrl: /avatars/10898c7886e25c64d58334083e3cf2d5.svg
paper_authors.fullname: Julian Spravil
paper_authors.type: user
paper_authors.user: Spravil
paper_authors._id: 67da8b794e1138ddc328de0a
paper_authors.name: Sebastian Houben
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:47:14.517Z
paper_authors._id: 654892f5c6dadd513f5a7143
paper_authors.avatarUrl: /avatars/c167b45776391e02709246670e4843d1.svg
paper_authors.fullname: Sebastian Houben
paper_authors.type: user
paper_authors.user: EthylS2
paper_authors._id: 67da8b794e1138ddc328de0b
paper_authors.name: Sven Behnke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:47:23.428Z
paper_authors._id: 65df1a033e759810359ad7c8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65df1a033e759810359ad7c8/51dnHqNxFuOT6pStpLYyX.jpeg
paper_authors.fullname: Sven Behnke
paper_authors.type: user
paper_authors.user: sven-behnke
paper_title: Florenz: Scaling Laws for Systematic Generalization in Vision-Language
    Models
paper_summary: Cross-lingual transfer enables vision-language models (VLMs) to perform
  vision tasks in various languages with training data only in one language.
  Current approaches rely on large pre-trained multilingual language models.
  However, they face the curse of multilinguality, sacrificing downstream task
  performance for multilingual capabilities, struggling with lexical ambiguities,
  and falling behind recent advances. In this work, we study the scaling laws of
  systematic generalization with monolingual VLMs for multilingual tasks,
  focusing on the impact of model size and seen training samples. We propose
  Florenz, a monolingual encoder-decoder VLM with 0.4B to 11.2B parameters
  combining the pre-trained VLM Florence-2 and the large language model Gemma-2.
  Florenz is trained with varying compute budgets on a synthetic dataset that
  features intentionally incomplete language coverage for image captioning, thus,
  testing generalization from the fully covered translation task. We show that
  not only does indirectly learning unseen task-language pairs adhere to a
  scaling law, but also that with our data generation pipeline and the proposed
  Florenz model family, image captioning abilities can emerge in a specific
  language even when only data for the translation task is available. Fine-tuning
  on a mix of downstream datasets yields competitive performance and demonstrates
  promising scaling trends in multimodal machine translation (Multi30K, CoMMuTE),
  lexical disambiguation (CoMMuTE), and image captioning (Multi30K, XM3600, COCO
  Karpathy).
paper_ai_summary: Monolingual vision-language models achieve competitive performance across multilingual tasks through selective training and data generation, demonstrating scaling trends in translation and captioning.
paper_ai_keywords: cross-lingual transfer vision-language models multilingual language models monolingual encoder-decoder VLMs systematic generalization pre-trained VLM large language model synthetic dataset image captioning scaling law fine-tuning multimodal machine translation lexical disambiguation