NeoBabel: A Multilingual Open Tower for Visual Generation
Text-to-image generation advancements have been predominantly
English-centric, creating barriers for non-English speakers and perpetuating
digital inequities. While existing systems rely on translation pipelines, these
introduce semantic drift, computational overhead, and cultural misalignment. We
introduce NeoBabel, a novel multilingual image generation framework that sets a
new Pareto frontier in performance, efficiency and inclusivity, supporting six
languages: English, Chinese, Dutch, French, Hindi, and Persian. The model is
trained using a combination of large-scale multilingual pretraining and
high-resolution instruction tuning. To evaluate its capabilities, we expand two
English-only benchmarks to multilingual equivalents: m-GenEval and m-DPG.
NeoBabel achieves state-of-the-art multilingual performance while retaining
strong English capability, scoring 0.75 on m-GenEval and 0.68 on m-DPG.
Notably, it performs on par with leading models on English tasks while
outperforming them by +0.11 and +0.09 on multilingual benchmarks, even though
these models are built on multilingual base LLMs. This demonstrates the
effectiveness of our targeted alignment training for preserving and extending
crosslingual generalization. We further introduce two new metrics to rigorously
assess multilingual alignment and robustness to code-mixed prompts. Notably,
NeoBabel matches or exceeds English-only models while being 2-4x smaller. We
release an open toolkit, including all code, model checkpoints, a curated
dataset of 124M multilingual text-image pairs, and standardized multilingual
evaluation protocols, to advance inclusive AI research. Our work demonstrates
that multilingual capability is not a trade-off but a catalyst for improved
robustness, efficiency, and cultural fidelity in generative AI.
686ec992d938c25d68441a4d
Mohammad Mahdi Derakhshani
686ec992d938c25d68441a4e
Dheeraj Varghese
686ec992d938c25d68441a4f
Marzieh Fadaee
686ec992d938c25d68441a50
Cees G. M. Snoek
NeoBabel: A Multilingual Open Tower for Visual Generation
Text-to-image generation advancements have been predominantly
English-centric, creating barriers for non-English speakers and perpetuating
digital inequities. While existing systems rely on translation pipelines, these
introduce semantic drift, computational overhead, and cultural misalignment. We
introduce NeoBabel, a novel multilingual image generation framework that sets a
new Pareto frontier in performance, efficiency and inclusivity, supporting six
languages: English, Chinese, Dutch, French, Hindi, and Persian. The model is
trained using a combination of large-scale multilingual pretraining and
high-resolution instruction tuning. To evaluate its capabilities, we expand two
English-only benchmarks to multilingual equivalents: m-GenEval and m-DPG.
NeoBabel achieves state-of-the-art multilingual performance while retaining
strong English capability, scoring 0.75 on m-GenEval and 0.68 on m-DPG.
Notably, it performs on par with leading models on English tasks while
outperforming them by +0.11 and +0.09 on multilingual benchmarks, even though
these models are built on multilingual base LLMs. This demonstrates the
effectiveness of our targeted alignment training for preserving and extending
crosslingual generalization. We further introduce two new metrics to rigorously
assess multilingual alignment and robustness to code-mixed prompts. Notably,
NeoBabel matches or exceeds English-only models while being 2-4x smaller. We
release an open toolkit, including all code, model checkpoints, a curated
dataset of 124M multilingual text-image pairs, and standardized multilingual
evaluation protocols, to advance inclusive AI research. Our work demonstrates
that multilingual capability is not a trade-off but a catalyst for improved
robustness, efficiency, and cultural fidelity in generative AI.
NeoBabel, a multilingual image generation framework, achieves state-of-the-art performance across six languages while maintaining efficiency and robustness, outperforming existing multilingual models.
multilingual image generation large-scale multilingual pretraining high-resolution instruction tuning m-GenEval m-DPG crosslingual generalization code-mixed prompts multilingual evaluation protocols