## Navigating Text-to-Image Generative Bias across Indic Languages

Surbhi Mittal 1 , Arnav Sudan , Mayank Vatsa 1 1 , Richa Singh 1 , Tamar Glaser 2 , and Tal Hassner 3

1 Department of CSE, IIT Jodhpur, Rajasthan, India 2 Meta, Menlo Park, California, USA 3 Weir P.B.C., Alameda, California, USA

Abstract. This research investigates biases in text-to-image (TTI) models for the Indic languages widely spoken across India. It evaluates and compares the generative performance and cultural relevance of leading TTI models in these languages against their performance in English. Using the proposed IndicTTI benchmark, we comprehensively assess the performance of 30 Indic languages with two open-source diffusion models and two commercial generation APIs. The primary objective of this benchmark is to evaluate the support for Indic languages in these models and identify areas needing improvement. Given the linguistic diversity of 30 languages spoken by over 1.4 billion people, this benchmark aims to provide a detailed and insightful analysis of TTI models' effectiveness within the Indic linguistic landscape. The data and code for the IndicTTI benchmark can be accessed at https://iabrubric.org/resources/other-databases/indictti .

Keywords: text-to-image generation · multilingual · bias

## 1 Introduction

Text-to-image (TTI) generative technologies have transformed the landscape of digital media, distinguished by their extraordinary ability to produce detailed and visually engaging images from written text [12,14]. These models have found widespread application in a variety of fields, including gaming, animation, architecture, and fashion, offering significant benefits in terms of enhancing creativity and streamlining production timelines. However, despite the expansion of access through open-source models, challenges persist in achieving linguistic inclusivity [15]. The datasets these models are trained on are predominantly composed of English-language content, derived from both proprietary sources [12] and internet scraping [14, 16]. This reliance on English-centric data adversely affects the quality of images generated from text in languages other than English.

As the integration of TTI models into various industries deepens, it becomes imperative to conduct a thorough evaluation of their performance and the diversity of the outputs they produce. While there has been research to study

Fig. 1: (Top) Images generated by Midjourney when given equivalent prompts in the English and Hindi languages highlighting the tendency of the model to generate incorrectly. (Bottom) Images generated by DallE-3, when given equivalent prompts in the English and Hindi languages, highlight astonishingly different cultural representations.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-4ccf0f56-c41e-4e54-b2f0-2c288e8f63fe.jpg)

biases in TTI models [5, 7], the current evaluation landscape is notably lacking in benchmarks capable of effectively measuring these models' performance across a diverse linguistic spectrum, often overlooking the rich variety of global languages [15]. To address this gap, we introduce the IndicTTI benchmark , an evaluation benchmark designed to explore and measure the biases present in text-to-image generation technologies, with an emphasis on a wider range of languages beyond those typically studied. This benchmark seeks to explore the linguistic diversity of a global population exceeding 1.4 billion, aiming not just to spotlight the shortcomings of existing TTI models but also to foster a broader application of these technologies that is truly inclusive and reflective of the myriad languages and cultures worldwide. Through this in-depth analysis, our goal is to spur improvements in TTI technologies, ensuring they cater to a wider audience and promote a more inclusive environment within digital media creation.

In this research, we examine the performance of existing models in 30 languages other than English, with a specific focus on Indic languages, which are spoken by over a billion people worldwide. We believe that this benchmark is a step toward analyzing TTI models for a rich and diverse culture with a large population [9]. The 30 languages are written in 10 different scripts and belong to multiple language families. Among these, we have included the Scheduled Languages, which are considered the major literary languages of India and have a significant volume of literature [9]. We study two aspects of TTI models for assessing bias

Table 1: The list of Indic languages included in the IndicTTI benchmark. † These languages are present in more than one script in the benchmark. *Languages not listed as part of Indic languages as per the Indian Constitution but spoken by millions.

| Assamese asm (Beng)   | Malayalam mal (Mlym)   | Bodo brx (Deva)    | Marathi mar (Deva)      | Bengali ben (Beng)        | Manipuri † mni (Beng, Mtei)   |
|-----------------------|------------------------|--------------------|-------------------------|---------------------------|-------------------------------|
| Odia ory (Orya)       | Konkani gom (Deva)     | Punjabi pan (Guru) | Tamil tam (Taml)        | Sanskrit san (Deva)       | Hindi hin (Deva)              |
| Telugu tel (Telu)     | Maithili mai (Deva)    | Urdu urd (Arab)    | Kannada kan (Knda)      | Gujarati guj (Gujr)       | Kashmiri † kas (Arab, Deva)   |
| Awadhi* awa (Deva)    | Bhojpuri* bho (Deva)   | Magahi* mag (Deva) | Sinhala* sin (Sinh) †   | Chhattisgarhi* hne (Deva) | Chhattisgarhi* hne (Deva)     |
| Dogri doi (Deva)      | Nepali npi (Deva)      | Santali sat (Olck) | Sindhi snd (Deva, Arab) |                           |                               |

towards Indic languages, (i) model performance and (ii) model representation. We identify issues of correctness in generation, the presence of cultural elements, and the differences between open-source and API-based models. We utilize Stable Diffusion [14] and Alt Diffusion [6] models as part of our open-source analysis, and Midjourney [1] and Dalle3 [12,13] as part of the API-based generative models. We propose the Cyclic Language-Grounded Correctness (CLGC) metric to evaluate model performance and the Self-Consistency Across Languages (SCAL) metric to evaluate representation bias in the model generations. We further evaluate the model performance through Language-Grounded Correctness (LGC) and Image-Grounded Correctness (IGC) metrics. Through our analysis, we observe how certain language scripts evoke imagery of Indian Gods in Midjourney, the overwhelmingly high generation of Asian women and couples in open-source models, and the high accuracy of generations in Dalle3 (Refer Fig. 1). Finally, we discuss the correlation between language script and cultural influences observed in the models.

## 2 Related Work

Limited studies have been conducted to benchmark the performance of TTI models across different languages [11,15]. In NLP-based tasks, MEGA [2] benchmarks generative Large Language Models (LLMs) across 16 NLP datasets covering 70 diverse languages. This evaluation compares the performance of generative LLMs like Chat-GPT and GPT-4. Similarly, with a specific focus on the Indic languages, in IndicTrans2 [8], the authors provide a transformer-based multilingual NMT model that supports high-quality translations across all the 22 scheduled Indic languages.

In the domain of text-to-image generative models, the recent Coco-crola benchmark [15] focuses on the evaluation of DallE and Stable Diffusion models for seven languages, namely English, Spanish, German, Chinese, Japanese, Hebrew, and Indonesian. Among these languages, English, German, and Indonesian share the Latin script, whereas Japanese and Chinese use the same script. Further, the authors focus on the generation of simple concepts through translation; however, in everyday applications, users are more likely to generate images through complex prompts. With this in mind, we focus on more complex

Fig. 2: Pipeline of the generation and evaluation of the IndicTTI benchmark. After generating images from four TTI models, their bias is measured across the parameters of their correctness performance as well as representational diversity. The metrics are computed through the use of high-level semantic features extracted from the generated images and their prompts.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-86d03d63-d3b5-4983-8e8e-01ece9935573.jpg)

prompts describing multiple elements in the generated image. The authors also propose the distinctiveness, coverage, and self-consistency measures. Very limited research has also been conducted on the design of multilingual text-to-image generative models [6,10]. One such model is the AltDiffusion model, which is a multilingual TTI diffusion model designed to process multiple languages [6]. Recently, Struppek et al. [17] have shown the influence of script-specific characters on the cultural aspects of the generated images.

In this work, we study the correctness and cultural representation bias of the TTI models, with a focus on Indic languages. However, the metrics and analysis in this study can be organically extended for the analysis of more languages.

## 3 Benchmark Design

In this work, we propose the IndicTTI benchmark, which comprises of 31 languages, evaluated with four text-to-image generative models with six evaluation metrics covering aspects of correctness and representation. We describe the various design components of the benchmark below.

## 3.1 Indic Languages and Prompts

For this benchmark, we utilize 30 Indic languages in addition to the English language. The languages are listed in Table 1. For prompts, we utilize the COCONLLB dataset [18,19], which contains image-caption pairs for over 500K images, along with captions translated into 200 languages. In previous work [15], the authors use simplistic concepts for prompts such as 'a photo of a plane." However, the expectation from these models is to yield accurate and unbiased results in

day-to-day usage. We believe that this expectation is better met with more complex prompts such as those present in the COCO-NLLB dataset. Therefore, we sample 1000 diverse image-caption pairs from the dataset. The dataset contains 24 Indic languages for which the captions are directly taken. These 24 languages do not include 4 languages that form a part of the 22 scheduled languages of India as per the Indian Constitution, namely Bodo, Dogri, Konkani, and Santali. Using the IndicTrans2 translator [8], we translate English captions to these languages along with two other languages, Manipuri (in Meitei script) and Sindhi (in Devanagari script). This leads to a total of 1000 prompts in 31 languages, including English, along with an associated image.

## 3.2 TTI Models and Generated Images

We utilize four different text-to-image models for the benchmark. For opensource models , we use the Stable Diffusion [14] and AltDiffusion Models [6]. For API-based models , we use Dalle3 [12] and Midjourney [1].

Using the open-source models, we generate 4 images per prompt for the selected 1000 prompts across 31 languages, leading to 124K images corresponding to Stable Diffusion and AltDiffusion each. For API-based models, we generate over a subset of 200 prompts for Midjourney and over 50 prompts for Dalle3. This leads to a total of 24.8K images for Midjourney and 6.2K images for Dalle3. The evaluation is done over these sets and additionally over a subset of 40 prompts common across all models. The Dalle3 and Midjourney APIs are paid, and further details about the generation process are provided in the supplementary.

## 4 Evaluation Methods

In the IndicTTI benchmark, we focus on two aspects of TTI model evaluationcorrectness and representation. Correctness refers to the ability to measure the semantic faithfulness with which the model generates the images for the corresponding prompt. On the other hand, representation refers to the measurement of the diversity of generated images within and across the different languages for the given prompts. The complete pipeline of the study is depicted in Fig. 2.

We adopt the following notation to describe the metrics and the evaluation pipeline. Let the set of languages be L . The set of prompts is denoted by P = { p , p 1 2 , ..p k } , and the corresponding representative images are Q = { q 1 , q 2 , ..q k } . For each prompt in P , we generate n images, resulting in a total of n P | || L | images for a given generative model g . The set of images generated by g for a prompt p k in language l ∈ L is denoted as,

$$I _ { p _ { k }, l } & = \{ i _ { p _ { k }, l, r } \} _ { r = 0 } ^ { n } & ( 1 )$$

where, i p k ,l,r denotes the image generated using g as g p ( k , l ) . Utilizing this notation, we define the metrics in this section. Across all the metrics, we compute high-level semantic features of images (using feature extractor f ) and text (using feature extractor t ) for effective comparison using the similarity function ϕ .

## 4.1 Correctness-based Metrics

We employ three correctness-based metrics in our study, including the proposed Cyclic Language-Grounded Correctness metric. The image-grounded and language-grounded correctness metrics are inspired by the correctness metric in CocoCrola [15]. These three metrics comprehensively evaluate the generated images across text-text, image-image, and image-text spaces. A higher correctness value indicates greater accuracy of the generative model in producing images that are faithful to the text.

Cyclic Language-Grounded Correctness (CLGC): This metric evaluates the generated images I in the text-text space. This is achieved in a cyclic manner through generating captions over I using a caption generator c , as,

$$p _ { k } ^ { \prime } & = c ( i _ { p _ { k }, l, r } )$$

where, p ′ k denotes the generated caption for r th image in set I p k ,l . Then, with p k and p ′ k as prompts in the English language for the language l , we obtain,

$$C L G C ( l ) = \frac { 1 } { n | P | } \sum _ { p _ { k } } ^ { P } \sum _ { r = 0 } ^ { n } \phi [ \, t ( p _ { k } ), \, t ( p _ { k } ^ { \prime } ) \, ]$$

Image-Grounded Correctness (IGC): For this metric, the correctness of the generated images is evaluated in the image-image space, as,

$$I G C ( l ) = \frac { 1 } { n | P | } \sum _ { p _ { k } } ^ { P } \sum _ { r = 0 } ^ { n } \phi [ \, f ( q _ { k } ), \, f ( i _ { p _ { k } } ) \, ]$$

where, the ground-truth image q k is compared to each of the n generated images for the prompt p k .

Language-Grounded Correctness (LGC): This metric evaluates correctness in the image-text space with p k denoting the prompt in the English language,

$$L G C ( l ) = \frac { 1 } { n | P | } \sum _ { p _ { k } } ^ { P } \sum _ { r = 0 } ^ { n } \phi [ \, t ( p _ { k } ), \, f ( i _ { p _ { k } } ) \, ]$$

## 4.2 Representation-based Metrics

In this section, we discuss the three metrics utilized for measuring the representativeness of concepts across the languages. We propose the Self-Consistency Across Languages metric to capture the bias in representation across the different languages. The other two metrics are inspired by the self-consistency and distinctiveness metrics in previous work [15].

Self-Consistency Across Languages (SCAL): This metric captures the variation in the generated images for the same prompt across the different languages.

So, for a language pair ( l a , l b ) ,

̸

$$S C A L ( l _ { a }, l _ { b }, p _ { k } ) = \frac { 1 } { ^ { n } C _ { 2 } } \sum _ { \substack { u = 0 \\ \lambda } } ^ { n } \sum _ { v = 0 } ^ { n } \phi [ \, f ( i _ { p _ { k }, l _ { a }, u } ), \, f ( i _ { p _ { k }, l _ { b }, v } ) \, ] ; u \neq v \quad \ \ ( 6 )$$

$$S C A L ( l _ { a }, l _ { b } ) = \frac { 1 } { | P | } \sum _ { p _ { k } } ^ { P } S C A L ( l _ { a }, l _ { b }, p _ { k } )$$

(8)

Then, the overall SCAL score for the given set of languages becomes,

̸

$$S C A L = \frac { 1 } { | L | C _ { 2 } } \sum _ { l _ { a } } ^ { L } \sum _ { l _ { b } } ^ { L } S C A L ( l _ { a }, l _ { b } ) ; \ a \neq b$$

The lower the value of SCAL, the less consistent the generations are across the different languages. On the other hand, a higher value of SCAL would demonstrate high consistency between concepts across the different languages.

Self-Consistency Within Language (SCWL): This metric is computed within the images generated for a particular prompt for a given language, as,

̸

$$S C W L ( l, p _ { k } ) = \frac { 1 } { ^ { n } C _ { 2 } } \sum _ { \substack { u = 0 \\ w = 0 } } ^ { n } \sum _ { v = 0 } ^ { n } \phi [ \, f ( i _ { p _ { k }, u } ), \, f ( i _ { p _ { k }, v } ) \, ] ; u \neq v \quad \quad ( 1 0 )$$

$$S C W L ( l ) & = \frac { 1 } { | P | } \sum _ { p _ { k } } ^ { P } S C W L ( l, p _ { k } ) & ( 1 1 )$$

Ahigh self-consistency showcases good consistency between the semantic content of the generated images.

Distinctiveness Within Language (DWL): This metric depicts the diversity across the images generated for the different prompts p a and p b ,

̸

$$S W L ( l ) = 1 - \frac { 1 } { | P | C _ { 2 } } \sum _ { p _ { a } } ^ { P } \sum _ { p _ { b } } ^ { P } \phi [ \, f ( i _ { p _ { a } } ), \, f ( i _ { p _ { b } } ) \, ] ; a \neq b \, \quad \ \ ( 1 2 )$$

where a higher distinctiveness score showcases the capability of the model to generate diverse images with varying prompts.

In addition to quantitative evaluation, we conduct an extensive qualitative evaluation of the generated images with respect to the captions, languages, and scripts. The implementation details are provided in the supplementary.

## 5 Benchmark Results and Analysis

In this section, we report results on the 4 models used for evaluation, namely Stable Diffusion, AltDiffusion, Midjourney, and Dalle3, on a common set of prompts.

| Stable Alt Diffusion Midjourney Dalle3   | Model            | Model Stable Diffusion Alt Diffusion Midjourney Dalle3   |
|------------------------------------------|------------------|----------------------------------------------------------|
| Diffusion 33.98 30.90 32.25 33.06        | en               | en 33.98 30.90 32.25 33.06                               |
| 5.85 7.72 3.99 30.59                     | urd (Arab)       | asm (Beng) 3.90 6.50 2.08 26.42                          |
| 3.80 5.41 3.19 23.09                     | 30.54 kas (Arab) | ben (Beng) 3.26 6.87 1.95                                |
| 4.12 7.22 3.33 28.30                     | 28.04 kas (Deva) | guj (Gujr) 3.28 7.34 2.86                                |
| 4.26 8.22 3.66 28.68                     | 31.57 mai (Deva) | hin (Deva) 3.79 7.56 3.27                                |
| 5.83 8.04 4.30 28.91                     | awa (Deva)       | kan (Knda) 4.21 10.67 4.03 28.48                         |
| 4.10 7.82 2.95 29.99                     | bho (Deva)       | mal (Mlym) 4.09 6.53 2.44 28.22                          |
| 4.25 7.70 4.35 30.02                     | hne (Deva)       | mar (Deva) 3.61 7.81 2.87 30.87                          |
| 4.45 7.81 3.31 29.39                     | mag (Deva)       | mni (Beng) 3.50 4.54 1.73 14.98                          |
| 3.50 5.99 1.78 22.86                     | sin (Sinh)       | npi (Deva) 3.68 8.25 3.36 29.17                          |
| 3.90 4.73 3.68 12.93                     | brx (Deva)       | ory (Orya) 3.56 8.57 3.00 26.67                          |
| 3.55 5.98 3.56 29.10                     | doi (Deva)       | pan (Guru) 3.01 7.87 2.59 31.41                          |
| 4.11 6.12 2.86 24.05                     | gom (Deva)       | san (Deva) 3.77 8.49 3.41 24.32                          |
| 3.71 5.76 2.45 2.09                      | sat (Olck)       | snd (Arab) 4.16 6.53 3.91 19.92                          |
| 4.11 6.82 3.32 27.70                     | snd (Deva)       | tam (Taml) 4.23 7.12 3.92 26.24                          |
| 4.52 6.30 2.64 1.52                      | mni (Mtei)       | tel (Telu) 4.09 6.00 2.76 26.68                          |

| Stable Diffusion Alt Diffusion   | Midjourney Dalle3   | Alt Diffusion   | Model Stable Diffusion   |
|----------------------------------|---------------------|-----------------|--------------------------|
| 51.21 46.58                      | 48.94 48.92         | 46.58           | en 51.21                 |
| (Arab) (Arab) 22.92 25.40        | 22.69 40.02 urd     | 24.23           | asm (Beng) 24.19         |
| 22.58 24.45                      | 22.66 44.22 kas     | 25.08           | ben (Beng) 23.84         |
| 23.41 25.01                      | 23.06 43.75 kas     | 25.18           | guj (Gujr) 23.25         |
| 23.71 26.39                      | 22.50 44.86 mai     | 25.43           | hin (Deva) 23.53         |
| (Deva) 25.16 25.97               | 22.81 42.74 awa     | 27.74           | kan (Knda) 23.24         |
| 22.98 25.53                      | 22.04 43.12 bho     | 24.27           | mal (Mlym) 22.96         |
| (Deva) 23.19 25.59               | 22.75 46.10 hne     | 25.42           | mar (Deva) 23.53         |
| 23.41 26.10                      | 21.97 30.91 mag     | 22.79           | mni (Beng) 23.58         |
| (Deva) (Sinh) 24.00 24.60        | 22.29 43.14 sin     | 25.98           | npi (Deva) 23.96         |
| 23.83 23.82                      | 22.82 40.00 brx     | 26.27           | ory (Orya) 23.76         |
| (Deva) (Deva) 22.90 24.87        |                     | 25.48 23.05     | pan (Guru) 22.97         |
| 23.88 24.73                      | 45.06 doi           | 25.68           |                          |
| (Deva) (Olck) 24.14 24.16        | 22.81 37.66 gom     |                 | san (Deva) 23.11         |
| (Deva) 23.64 25.16               | 22.55 41.25 snd     | 25.06           | snd (Arab) 23.06         |
|                                  |                     |                 | tam (Taml) 22.68         |
| (Mtei) 23.98 24.35               | 22.09 41.39 mni     | 22.19           | tel (Telu)               |
|                                  |                     |                 | 22.61                    |

models provide high correctness for English languages while providing lower values for Indic languages.

| Alt Diffusion Midjourney Dalle3   | Model Stable Diffusion       | Model Stable Diffusion Alt Diffusion Midjourney Dalle3   |
|-----------------------------------|------------------------------|----------------------------------------------------------|
| 64.53 58.16 64.31 65.70           | 65.70 en                     | en 64.53 58.16 64.31                                     |
| 10.08 17.68 15.38 64.02           | 12.99 53.36 urd (Arab)       | asm (Beng) 7.29 14.65                                    |
| 6.29 15.96 12.72 47.45            | 11.89 60.73 kas (Arab)       | ben (Beng) 6.39 16.70                                    |
| 7.52 15.46 12.37 55.51            | 12.49 57.25 kas (Deva)       | guj (Gujr) 7.24 15.97                                    |
| 7.13 16.93 11.92 58.40            | 64.16 mai (Deva)             | hin (Deva) 7.35 16.79 11.31                              |
| 10.11 16.32 13.04 57.37           | 12.12 60.13 awa (Deva)       | kan (Knda) 7.13 22.98                                    |
| 7.12 15.41 11.47 60.75            | 57.68 bho (Deva)             | mal (Mlym) 6.91 16.08 10.92                              |
| 8.25 16.74 12.06 59.18            | 61.84 hne (Deva)             | mar (Deva) 7.53 16.50 11.99                              |
| 7.66 16.80 12.53 60.42            | 9.80 35.24 mag (Deva)        | mni (Beng) 6.50 12.17                                    |
| 6.25 13.04 11.96 49.03            | 59.90 sin (Sinh)             | npi (Deva) 8.03 17.97 11.89                              |
| 6.26 11.98 12.34 28.87            | 18.82 12.56 53.96 brx (Deva) | ory (Orya) 8.36                                          |
| 7.95 13.50 11.77 59.84            | 62.67 doi (Deva)             | pan (Guru) 8.78 17.64 9.32                               |
| 7.78 15.82 12.59 50.18            | 17.99 12.19 51.22 gom (Deva) | san (Deva) 8.14                                          |
| 10.51 14.13 11.36 10.93           | 45.19 sat (Olck)             | snd (Arab) 11.29 14.82 13.49                             |
| 7.59 13.37 12.35 56.78            | 12.84 53.98 snd (Deva)       | tam (Taml) 7.30 18.10                                    |
| 10.17 15.25 9.72 11.19            | 12.16 55.26 mni (Mtei)       | tel (Telu) 7.65 14.50                                    |

Table 2:

Cyclic Language Grounded Correctness (CLGC) (%) across the different Indic languages in the IndicTTI benchmark. Existing

We provide the results on the complete set of prompts for the models in the supplementary. The observations are consistent with those made on the common subset reported in this section.

Correctness Bias: As discussed in the previous section, we evaluate the correctness of generation across the Indic languages through the metrics of CLGC, IGC, and LGC, which correspond to the text-text, image-image, and image-text spaces, respectively. In the text-text space, we observe that the CLGC metric is significantly high for the English language (Refer Table 2). This demonstrates that the generated images have high faithfulness to the text, as the captions generated for the generated images are semantically closest to the original English caption. All models have a high CLGC value of &gt; 60% for the English language. On the other hand, CLGC values drop significantly for Indic languages for Stable Diffusion and AltDiffusion models, with the performance dropping to 7.29% from 64.53% and 14.65% from 58.16%, respectively for the Assamese language. The values of CLGC tend to be lower for languages with a Bengali script, namely Manipuri (Beng), Assamese, and Bengali. On computing the average across the 30 Indic languages and comparing the CLGC scores with those obtained for the English language, there is a 56.64% drop in performance for the Stable Diffusion model. Similarly, there is a 42.16%, 52.16%, and 13.62% drop for AltDiffusion, Midjourney, and Dalle3, with Dalle3 providing the best performance.

Similarly, in the image-image space, we observe higher IGC metric values for the English language, showcasing up to 50% image semantic similarity with the ground truth images. With Indic languages, there is an average drop of 27.74% and 21.53% for the Stable Diffusion and AltDiffusion models. While there is a high drop of 26.32% for Midjourney generated images, the drop is significantly lesser for Dalle3 at 8.70%, signifying a superior understanding of Indic languages. For image-text similarity, we use the English language as the prompt of comparison. We observe higher LGC metric values for the images generated using English prompts, in the range of 30-35% across the models. These values drop sharply for Stable Diffusion, AltDiffusion, and Midjourney, with average LGC values of 4.01%, 7.08%, and 3.12% across the Indic languages, respectively. Dalle3 has the highest LGC value at 25.09%, with extremely low values observed for the Manipuri (Meitei script) (1.52%), Santali (2.09%), and Bodo (12.93%) languages.

Across all the metrics, we observe that Dalle3 outperforms all other models when evaluated for Indic languages with a significant margin. However, Dalle3 struggles with the Manipuri (Meitei script), Santali, and Bodo languages. Other models struggle to generalize on the Indic languages with AltDiffusion performing the best among the other three models. This behavior could be a result of increased generalizability due to the multilingual training of the model. On the other hand, while all models perform similarly for the English language, AltDiffusion performs the worst, possibly due to catastrophic forgetting when trained for multilinguality.

Representativeness Bias: To better understand the representative capacity of the models, we evaluate the SCAL, SCWL, and DWL metrics as explained in

Fig. 3: Performance of the SCAL metric for (a) Stable Diffusion, (b) Alt Diffusion, (c) Midjourney, and (d) Dalle3 models, demonstrating the extent of similarity between the generated images for the same prompt across the different language pairs.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-687d3f4a-1630-4c6f-9062-f83be594ac67.jpg)

the previous section. While these metrics do not contain the capability to estimate the correctness of generation on their own, their values showcase important behavior in terms of diversity in model generation.

The SCAL metric evaluates the self-consistency of generated images across the different prompts. A high SCAL value represents the model's ability to consistently generate similar semantic content across languages. In Fig. 3, we plot the SCAL values across the different language pairs for the four models. From Fig. 3(a), we observe that there is little semantic similarity of concepts between the English language and other languages. Since we know that these models are primarily trained to perform well on English, this is also an indicator of poor performance across the other languages. Interestingly, the concepts generated across other languages have higher overlap showing that they generate images that are far similar in semantic content, despite being possibly incorrect in their generations. In the previous section, we observed that the performance of generation for Indic languages is only high in Dalle3. This is demonstrated well in Fig. 3(d) where we observe that the semantic similarity between English and Indic languages is high, with performance being poorest between Santali and Manipuri (Meitei script), followed by Bodo and Manipuri (Bengali script).

The overall SCAL metric value as per Eqn. 9 for the Stable Diffusion, AltDiffusion, Midjourney, and Dalle3 models is 25.44%, 23.73%, 26.75%, and 29.90%, respectively. This indicates an overall high consistency of Dalle3 in generating concepts across different languages. It is also evident from Fig. 3 that the Stable

Fig. 4: Performance for the (a) SCWL and (b) DWL metrics of the benchmark showcasing self-consistency and distinctiveness of concepts within language, respectively.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-65353280-a9be-4dba-8f29-be184833f352.jpg)

Diffusion, AltDiffusion, and Midjourney generators suffer greatly in maintaining semantic consistency with images generated via English, whereas they tend to generate similar content for other languages. From Fig. 3(c), we also observe that Midjourney has high semantic consistency between images generated from languages with the Devanagari script, highlighting the language script plays a crucial part in the semantic content of the image.

Next, we report the representativeness of different prompts within the same language through the SCWL metric . In Fig. 4(a), we observe that the selfconsistency of models is higher for the English language. A higher value demonstrates a more stable representation of concepts in the model. For Dalle3, the SCWL values for the different languages are comparable to those of English but still lower. In agreement with previous observations, the values are especially poor for the Santali, Manipuri (Meitei script), Bodo, and Manipuri (Bengali script) languages. In Midjourney, the values stay consistent, with the exception of the Santali and Manipuri (Meitei) languages. Similar trends are observed for the other models. When compared with other metrics, there is a low variation in the values of self-consistency within the language for the different models and languages. This behavior could be the result of the relative stability of the model in its generation. While the model may generate incorrectly, it associates a given text for generation in the latent space consistently. Therefore, the model may demonstrate high self-consistency by generating images that are related to a given prompt, albeit incorrectly.

Finally, in Fig. 4(b), results are presented for the DWL metric . The DWL metric showcases how distinct are the generations corresponding to the different prompts. This metric is dependent on the relatedness of the prompts used for generation. While sampling prompts for the IndicTTI benchmark, we ensured semantic diversity in the selected 1000 prompts. The details of the selection process are provided in the supplementary. It is evident from Fig. 4(b) that Dalle3 provides the highest level of distinctiveness between the generated prompts across the different languages, only suffering for the Santali and Manipuri (Meitei script) languages. Surprisingly, Midjourney has the lowest distinctiveness in its generations across the Indic languages, indicating it generates

Fig. 5: Showcasing the low semantic relevance of images generated by Stable Diffusion (right) and Alt Diffusion (left) models corresponding to prompts in Indic languages.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-ee797078-80c7-486d-86a1-9e9a944d4863.jpg)

similar content over the different prompts. Similarly, the Stable Diffusion model performs poorly. The AltDiffusion model provides the second highest distinctiveness among the four models. This could be attributed to its multilingual training, making it sensitive to different scripts during generation.

## 6 Qualitative Analysis

In this section, we qualitatively analyze the images generated by the different models across the different languages. Based on our observations in the previous section, we explore the generated images and study the alignment between the quantified metrics and generated images.

As in the previous section, we study two aspects of bias, correctness-based as well as representation-based. In Fig. 5, we observe that the Stable Diffusion and AltDiffusion models suffer greatly when prompted with Indic languages. The images generated using the prompts in Indic languages often do not resemble the description of the actual prompts. For example, when prompted to generate a swan in the Punjabi language, the Stable Diffusion model generates an image of 3 men in turbans, whereas the AltDiffusion model generates the image of a couple. Here again, the man is observed to be wearing a turban. We further observe a relatively high depiction of people in images, specifically those of Indian ethnicity, even when the actual content of the prompt does not involve any mention of people. Similarly, the lack of correspondence between the actual prompt and the generated images for Midjourney can be seen in Fig. 1. For example, on prompting Midjourney to generate images of 'a man in a brown jacket," it generates stereotypically Indian women. Other simplistic prompts in the Hindi language also lead to incorrect generations.

For Dalle3, we observe that the model generates correct images, relevant to the content of the prompt. This observation is corroborated by the high performance of Dalle3 on the metrics proposed with the benchmark. However, the Dalle3 model struggles with certain languages, such as Manipuri, Santali, and Bodo. In order to better understand the poor quality of generations, we prompted

Fig. 6: Showcasing how Dalle3 with ChatGPT4 creates images with cultural aspects for unsupported Indic languages after identifying prompts in (a) Manipuri (Meitei script), (b) Bodo, and (c) Santali.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-faacc1e7-3876-4ad7-8e65-efe73de0ce00.jpg)

the ChatGPT model, which uses Dalle3 for image generation, to generate some images for these languages. On providing the ChatGPT model with prompts in these languages, it seemed that the model did not understand these scripts at all. Specifically, it identified the script/language it was being prompted with and then continued to 'imagine" an image based on cultural influences. Some of these examples are shown in Fig. 6 where the model generates images influenced by the Manipuri, Nepali, and Santal cultures.

On observing the images for the different models (Refer Fig. 7), we notice a correlation between different language scripts and the images generated corresponding to them. All models, except Dalle3, frequently generate images of females and Indian religious figures such as Gods and pandits . Dalle3, on the other hand, tends to incorporate common Indian religious sites like temples and ghats . In Hindu culture, many women regularly wear sarees and bindi , and these elements are prominently featured in the images produced by both Stable Diffusion and AltDiffusion. Additionally, the Arabic script is frequently associated with elements of Muslim culture, such as the burqa and niqab . Elements of South Indian culture are often present in the images generated for prompts in Malayalam, a language widely spoken in Southern India. These correlations between script and generated images are common, especially in those produced by Midjourney. When prompted in Sanskrit, the sacred language of Hinduism and predominant in religious texts, the Stable Diffusion model is observed to generate images of Indian Gods. Similarly, prompts in Punjabi often result in images of men wearing turbans, reflecting the cultural and religious practices of Punjabi speakers. More observations are detailed in the supplementary materials.

## 7 Conclusion

In this research, we introduce the IndicTTI benchmark, a comprehensive framework that integrates a variety of prompts, several text-to-image (TTI) models, and an extensive assessment of generative biases using the proposed evaluation methods. Employing both performance-oriented and representativeness-centric

Fig. 7: Showcasing the influence of language scripts on the cultural aspects in Stable Diffusion (row 1), AltDiffusion (row 2), Midjourney (row 3), and Dalle3 (row 4).

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-41dff5ab-8f4c-4139-a990-35511130bd68.jpg)

metrics, we are able to thoroughly assess the capacity of the models for multilingual generation. The qualitative analysis further emphasizes the extent of cultural influences embedded within the models' outputs. While the presence of cultural elements is not inherently detrimental, some generated images may inadvertently reinforce cultural stereotypes. The IndicTTI benchmark makes a dual contribution: firstly, it establishes a robust quantitative standard for assessing the multilingual capabilities of TTI models; secondly, it highlights the incorporation of Indian cultural motifs in the imagery produced.

Limitations: In this paper, we conduct both quantitative and qualitative evaluations of various aspects of TTI systems in a multilingual context. The prompts used in our benchmark are primarily derived from COCO-NLLB, which translates English captions from the COCO dataset into multiple languages, or translated from English using the open-source IndicTrans2 model. Therefore, the generation performance for Indic prompts is indirectly affected by the quality of these translations. A user study, detailed in the supplementary materials, briefly examines the translation quality and demonstrates the appropriateness and accuracy of the translated captions. While the high performance of the Dalle3 model suggests that the translations are meaningful, it is essential to recognize the influence of translation in the benchmarking process. Furthermore, the semantic similarity between images and text is evaluated using large pre-trained models, which may possess inherent biases that could inadvertently impact the evaluation. Our comprehensive qualitative assessment generally supports the quantitative results derived from these pre-trained models. Nonetheless, there remains a possibility that inadvertent biases could affect the outcomes.

## Acknowledgements

S. Mittal is partially supported by the UGC-Net JRF Fellowship and IBM fellowship. M. Vatsa was partially supported through the Swarnajayanti Fellowship. This work was partially supported by Facebook AI. All data was stored, and experiments were performed on IITJ servers by IITJ faculty and students.

## References

- 1. Midjourney. https://www.midjourney.com/home (2024)
- 3. Agirre et al., E.: Semeval-2012 task 6: A pilot on semantic textual similarity*. In: International Workshop on Semantic Evaluation. pp. 7-8 (2012)
- 2. Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A.U., Ganu, T., Segal, S., Ahmed, M., Bali, K., Sitaram, S.: MEGA: multilingual evaluation of generative AI. In: Conference on Empirical Methods in Natural Language Processing. pp. 4232-4267. Association for Computational Linguistics (2023)
- 4. Costa-jussà et al., M.R.: No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672 (2022)
- 6. Chen, Z., Liu, G., Zhang, B., Yang, Q., Wu, L.: Altclip: Altering the language encoder in CLIP for extended language capabilities. In: Findings of the Association for Computational Linguistics: ACL. pp. 8666-8682. Association for Computational Linguistics (2023)
- 5. Chen, T., Hirota, Y., Otani, M., Garcia, N., Nakashima, Y.: Would deep generative models amplify bias in future models? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10833-10843 (2024)
- 7. Friedrich, F., Hämmerl, K., Schramowski, P., Libovicky, J., Kersting, K., Fraser, A.: Multilingual text-to-image generation magnifies gender stereotypes and prompt engineering may not help you. arXiv preprint arXiv:2401.16092 (2024)
- 8. Gala, J., Chitale, P.A., Raghavan, A.K., Gumma, V., Doddapaneni, S., M, A.K., Nawale, J.A., Sujatha, A., Puduppully, R., Raghavan, V., Kumar, P., Khapra, M.M., Dabre, R., Kunchukuttan, A.: Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. Transactions on Machine Learning Research (2023), https://openreview.net/forum?id= vfT4YuzAYA
- 9. Govt. of India: Indian languages. https://www.education.gov.in/sites/upload\_ files/mhrd/files/upload\_document/languagebr.pdf (2024)
- 11. Liu, B., Wang, L., Lyu, C., Zhang, Y., Su, J., Shi, S., Tu, Z.: On the cultural gap in text-to-image generation. arXiv preprint arXiv:2307.02971 (2023)
- 10. Li, Y., Chang, C., Rawls, S., Vulic, I., Korhonen, A.: Translation-enhanced multilingual text-to-image generation. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL. pp. 9174-9193. Association for Computational Linguistics (2023)
- 12. OpenAI: Dalle3. https://openai.com/dall-e-3 (2024)
- 14. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684-10695 (2022)
- 13. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International Conference on Machine Learning. pp. 8821-8831. PMLR (2021)

## 16

- 15. Saxon, M., Wang, W.Y.: Multilingual conceptual coverage in text-to-image models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL. pp. 4831-4848. Association for Computational Linguistics (2023)
- 17. Struppek, L., Hintersdorf, D., Friedrich, F., Schramowski, P., Kersting, K., et al.: Exploiting cultural biases via homoglyphs in text-to-image synthesis. Journal of Artificial Intelligence Research 78 , 1017-1068 (2023)
- 16. Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open largescale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35 , 25278-25294 (2022)
- 18. Visheratin, A.: Nllb-clip-train performant multilingual image retrieval model on a budget. arXiv preprint arXiv:2309.01859 (2023)
- 19. Visheratin, A.: Laion-coco-nllb. https : / / huggingface . co / datasets / visheratin/laion-coco-nllb (2024)

## A Benchmark Design

In this section, we provide additional details about the benchmark design.

## A.1 Indic Languages and Prompts

In this research, we introduce the IndicTTI benchmark where we study the performance of popular text-to-image (TTI) models in 30 languages. While the benchmark may be extended for any number of languages, we select 30 Indic languages, which are written in 10 different scripts and have roots in multiple language families. Detailed information about the different languages is provided in Table 5.

Table 5: The language family, script, language subfamilies, and number of native speakers for the 30 Indic languages in the IndicTTI benchmark. * represents the nonavailability of official reports regarding the statistics.

| Language Code                                                                                                                                                                                                                                                                 | Name                                                                                                                                                                                                               | Family                                                                                                                                                                                                                                                                                                      | Script                                                                                                                                                                                                                                                                                   | Sub-family                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | #Native Speakers                                                                                                                             |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|
| asm_Beng awa_Deva ben_Beng bho_Deva brx_Deva doi_Deva gom_Deva guj_Gujr hin_Deva hne_Deva kan_Knda kas_Arab kas_Deva mag_Deva mai_Deva mal_Mlym mar_Deva mni_Beng mni_Mtei npi_Deva ory_Orya pan_Guru san_Deva sat_Olck sin_Sinh snd_Arab snd_Deva tam_Taml tel_Telu urd_Arab | Assamese Awadhi Bengali Bhojpuri Bodo Dogri Konkani Gujarati Hindi Chhattisgarhi Kannada Kashmiri Magahi Maithili Malayalam Marathi Manipuri Nepali Odia Punjabi Sanskrit Santali Sinhala Sindhi Tamil Telugu Urdu | Indo-Aryan Indo-Aryan Indo-Aryan Indo-Aryan Sino-Tibetan Indo-Aryan Indo-Aryan Indo-Aryan Indo-Aryan Indo-Aryan Dravidian Indo-Aryan Indo-Aryan Indo-Aryan Dravidian Indo-Aryan Sino-Tibetan Indo-Aryan Indo-Aryan Indo-Aryan Indo-Aryan Austroasiatic Indo-Aryan Indo-Aryan Dravidian Dravidian Indo-Aryan | Bengali Devanagari Bengali Devanagari Devanagari Devanagari Devanagari Gujarati Devanagari Devanagari Kannada Perso-Arabic Devanagari Devanagari Devanagari Malayalam Devanagari Bengali Meitei Devanagari Odia Gurmukhi Devanagari Ol Chiki Sinhala Arabic Devanagari Tamil Telugu Urdu | Eastern Indo-Aryan Northern Indo-Aryan Eastern Indo-Aryan Northern Indo-Aryan Boroic Northern Indo-Aryan Southern Indo-Aryan Western Indo-Aryan Central Indo-Aryan Northern Indo-Aryan South Dravidian Northern Indo-Aryan Indo-Aryan Eastern Indo-Aryan Southern Dravidian Southern Indo-Aryan Central Tibeto-Burman Northern Indo-Aryan Eastern Indo-Aryan North Western Indo-Aryan Indo-Aryan Munda Indo-Aryan North Western Indo-Aryan South Dravidian South Central Dravidian Central Indo-Aryan | 15.3M 2.52M 97.2M * 1.4M 2.5M 2.2M 55.4M 528.3M 13M 43.7M 6.7M * 13.5M 34.8M 83.0M 1.7M 2.9M 37.5M 33.1M 0.02M 7.3M * 2.7M 69.0M 81.1M 50.7M |

For prompts, we utilize the COCO-NLLB dataset [18, 19], which contains image-caption pairs for over 500K images, along with captions translated into 200 languages. We sample 1000 diverse image-caption pairs from the dataset. In order to avoid prompts with proper nouns, such as names of celebrities and/or brand names, we filtered the dataset to remove any captions that contained capitalized words in the middle of the sentence. From the filtered dataset, we randomly subsample 1000 captions. Next, for a diverse selection, we computed sentence-level embeddings for the 1000 prompts using a SentenceFormer model and calculated the average similarity between any two prompts. This experiment was repeated for 1000 iterations, and the subset with the lowest sentence

Fig. 8: Plot showcasing the quality of captions per language.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-cae41242-bd80-49bd-898b-b0f02631b2dd.jpg)

similarity between prompts was selected. The subsets of 200 and 50 prompts for Midjourney and Dalle3, respectively, were chosen randomly from the selected subset of 1000 prompts.

## A.2 TTI Models and Generated Images

We utilize four different text-to-image models for the benchmark. For opensource models , we use the Stable Diffusion and AltDiffusion Models (without safety filters). In the main paper, we report results on the m2 variant of AltDiffusion trained on the English and Chinese languages. Extended results, including the m9 variant trained on the English, Chinese, Spanish, French, Russian, Japanese, Korean, Arabic, and Italian languages, are reported in the supplementary. The extended results exhibit a similar pattern to the m2 variant. Detailed results are provided in the subsequent sections. Unless specified otherwise, the AltDiffusion model refers to the m2 variant in this work. For API-based models , we use Dalle3 [12] and Midjourney [1], which are both paid. Stable Diffusion and AltDiffusion models generate images of size 512 x 512, whereas Midjourney and Dalle3 generate images of size 1024 x 1024.

## A.3 Quality of Translated Captions

Our prompts were primarily sourced from COCO-NLLB or translated from English using the IndicTrans2 model. As specified in the Limitations section of the main paper, this suggests that the translation quality may have influenced the performance of image generation with Indic prompts. To evaluate the translation quality, we conducted a user study on a common set of 40 prompts across 30 languages, validated by 65 annotators with an average proficiency of 4.78 out of 5 in at least one Indic language. We followed the XSTS protocol [3], also used to assess the translation quality of the NLLB model [4]. On a scale from 1 (worst) to 5 (best), the average translation quality for all languages is reported in Fig. 8, indicating that the sentences are mostly equivalent or paraphrases of each other, according to the XSTS protocol, and thus suitable for image generation.

In this experiment, the inter-rater agreement, measured through percent agreement, was 74.8%. For languages such as kas\_Deva, snd\_Deva, and sin\_Sinh, the translation quality of certain prompts was observed to be less than 3 on the XSTS scale.

## B Implementation Details

In this section, we discuss the implementation details involved in the creation of the IndicTTI benchmark as well as its evaluation.

For generation using open-source TTI systems of Stable Diffusion 4 and AltDiffusion 5 , we utilized the models available at HuggingFace. Dalle3 API was accessed through a Python script whereas the generation for Midjourney was done using Discord. For Stable Diffusion and AltDiffusion, the generation was done using an NVIDIA DGX Station consisting of 4 NVIDIA V 100 GPU with 32 GB VRAM each, using a batch size of 8. The generation was repeated using 4 seeds, providing 4 images for every prompt. The DDIM Scheduler was used for inferencing with 50 steps.

For evaluation, all experiments are conducted on LINUX-based systems using Python-based libraries, and specifically, the PyTorch library is used. To extract rich semantic text and image features for the evaluation metrics, we utilize various modules of the BLIP-2 model . We utilize the image encoder of the 6 BLIP-2 model as the image feature extractor f for computing the CLGC, IGC, SCAL, SCWL, and DWL metrics. Additionally, for the CLGC metric, we utilize the image-captioning capabilities of BLIP-2 captioner c to generate captions for generated images, and for extracting rich textual features, the SentenceFormer model . For the LGC metric, we require image-text features that are 7 extracted from BLIP-2 using the LAVIS 8 library. The similarity function ϕ is computed using the cosine similarity. The code for generation, as well as evaluation, can be accessed through https://iab-rubric.org/resources/otherdatabases/indictti .

## C Benchmark Results and Analysis

In this section, we report extended results on the common set and complete set of prompts. The observations are consistent with those reported in the main paper on the common subset.

In the correctness-based metrics , the CLGC, IGC, and LGC metrics over the common set of prompts are reported in Tables 6, 7, and 8, respectively. Similarly, the results for the three metrics on the complete set of prompts are reported

4 https://huggingface.co/runwayml/stable-diffusion-v1-5

5 https://huggingface.co/BAAI/AltDiffusion , https://huggingface.co/BAAI/ AltDiffusion-m9

6 https://huggingface.co/Salesforce/blip2-opt-2.7b

7 https://huggingface.co/sentence-transformers/all-mpnet-base-v2

8 https://github.com/salesforce/LAVIS/

Dalle3

Alt Diffusion

Midjourney

32.25

Stable Diffusion

Model

33.98

30.90

Alt Diffusion m9 27.56

33.06

3.99

7.72

10.38

30.59

3.19

5.85

3.80

5.41

6.25

23.09

3.33

7.22

10.74

28.30

3.66

4.12

Model

Stable Diffusion

Alt Diffusion

Midjourney

Dalle3

en

32.25

33.06

urd

30.90

33.98

Alt Diffusion m9 27.56

3.90

6.50

7.28

2.08

26.42

kas

1.95

30.54

kas

4.26

(Arab) (Arab) (Deva) (Deva)

6.87

10.92

2.86

28.04

mai

8.22

10.44

28.68

4.30

8.04

12.38

28.91

2.95

3.26

3.28

7.34

12.26

3.27

31.57

awa

(Deva)

5.83

4.10

7.56

12.58

4.03

28.48

bho

(Deva)

7.82

10.67

29.99

4.35

7.70

12.58

30.02

3.31

4.25

3.79

4.21

10.67

12.36

2.44

28.22

hne

4.45

(Deva) (Deva)

6.53

11.58

2.87

30.87

mag

7.81

12.56

29.39

1.78

5.99

11.04

22.86

3.68

4.09

3.61

7.81

11.37

4.54

3.66

1.73

14.98

sin

(Sinh)

3.50

3.90

4.73

4.71

3.55

5.98

12.93

9.69

3.56

29.10

8.20

2.86

24.05

2.09

29.17

3.36

brx

3.50

3.68

8.25

14.01

3.00

26.67

doi

4.11

6.12

5.16

3.56

8.57

11.73

2.59

31.41

gom

(Deva) (Deva) (Deva)

3.71

7.87

3.01

10.20

9.93

3.77

8.49

3.41

24.32

sat

(Olck)

5.76

4.11

2.45

6.82

9.91

4.52

3.32

6.30

27.70

5.65

2.64

1.52

6.53

7.61

3.91

19.92

snd

3.92

26.24

mni

4.16

4.23

7.12

11.39

2.76

26.68

4.09

6.00

11.30

(Deva) (Mtei)

en asm

ben guj

hin kan

mal mar

mni npi

ory pan

san snd

tam tel

(Beng) (Beng) (Gujr) (Deva) (Knda) (Mlym) (Deva) (Beng) (Deva) (Orya) (Guru) (Deva) (Arab) (Taml) (Telu)

Table 8:

Language-Grounded Correctness (LGC) (%) across the different Indic languages in IndicTTI on the common set of prompts.

| Midjourney Dalle3   | Stable Diffusion Alt Diffusion Alt Diffusion m9   | Model      | Alt Diffusion Midjourney Dalle3   | Alt Diffusion m9   | Model Stable Diffusion   |
|---------------------|---------------------------------------------------|------------|-----------------------------------|--------------------|--------------------------|
| 48.94 48.92         | 51.21 46.58 42.43                                 | en         | 42.43 48.94 48.92                 | 51.21 46.58        | en                       |
| 23.30 44.10         | 22.92 25.40 28.39                                 | urd (Arab) | 26.50 22.69 40.02                 | 24.19 24.23        | asm (Beng)               |
| 22.76 38.81         | 22.58 24.45 25.05                                 | kas (Arab) | 28.39 22.66 44.22                 | 23.84 25.08        | ben (Beng)               |
| 22.63 44.04         | 23.41 25.01 27.48                                 | kas (Deva) | 30.46 23.06 43.75                 | 23.25 25.18        | guj (Gujr)               |
| 22.90 43.18         | 23.71 26.39 27.34                                 | mai (Deva) | 28.94 22.50 44.86                 | 23.53 25.43        | hin (Deva)               |
| 23.09 44.88         | 25.16 25.97 29.31                                 | awa (Deva) | 29.74 22.81 42.74                 | 23.24 27.74        | kan (Knda)               |
| 22.65 44.22         | 22.98 25.53 28.24                                 | bho (Deva) | 29.59 22.04 43.12                 | 22.96 24.27        | mal (Mlym)               |
| 22.97 44.03         | 23.19 25.59 30.17                                 | hne (Deva) | 28.86 22.75 46.10                 | 23.53 25.42        | mar (Deva)               |
| 22.64 44.17         | 23.41 26.10 29.53                                 | mag (Deva) | 23.29 21.97 30.91                 | 23.58 22.79        | mni (Beng)               |
| 22.16 37.46         | 24.00 24.60 29.79                                 | sin (Sinh) | 30.67 22.29 43.14                 | 23.96 25.98        | npi (Deva)               |
| 22.57 29.76         | 23.83 23.82 23.28                                 | brx (Deva) | 29.55 22.82 40.00                 | 23.76 26.27        | ory (Orya)               |
| 22.90 44.69         | 22.90 24.87 27.16                                 | doi (Deva) | 28.76 23.05 45.06                 | 22.97 25.48        | pan (Guru)               |
| 22.79 38.91         | 23.88 24.73 25.83                                 | gom (Deva) | 26.54 22.81 37.66                 | 23.11 25.68        | san (Deva)               |
| 21.25 23.18         | 24.14 24.16 24.10                                 | sat (Olck) | 25.54 23.34 36.30                 | 23.06 24.68        | snd (Arab)               |
| 22.53 42.18         | 23.64 25.16 28.24                                 | snd (Deva) | 28.29 22.55 41.25                 | 22.68 25.06        | tam (Taml)               |
| 21.99 22.46         | 23.98 24.35 24.26                                 | mni (Mtei) | 28.62 22.09 41.39                 | 22.61 22.19        | tel (Telu)               |

Table 7:

Image-Grounded Correctness (IGC) (%) across the different Indic languages in IndicTTI on the common set of prompts.

| Midjourney Dalle3   | Alt Diffusion Alt Diffusion m9 53.19   | Model Stable Diffusion             | Model Stable Diffusion Alt Diffusion Alt Diffusion m9 Midjourney Dalle3   |
|---------------------|----------------------------------------|------------------------------------|---------------------------------------------------------------------------|
| 64.31 65.70         | en 64.53 58.16                         | 58.16 53.19 64.31 65.70            | en 64.53                                                                  |
| 15.38 64.02         | (Arab) 10.08 17.68 26.29               | 7.29 14.65 18.02 12.99 53.36 urd   | asm (Beng)                                                                |
| 12.72 47.45         | (Arab) 6.29 15.96 17.33                | 6.39 16.70 27.08 11.89 60.73 kas   | ben (Beng)                                                                |
| 12.37 55.51         | (Deva) 7.52 15.46 26.03                | 7.24 15.97 26.50 12.49 57.25 kas   | guj (Gujr)                                                                |
| 11.92 58.40         | (Deva) 7.13 16.93 23.69                | 7.35 16.79 29.78 11.31 64.16 mai   | hin (Deva)                                                                |
| 13.04 57.37         | (Deva) 10.11 16.32 28.06               | 7.13 22.98 26.66 12.12 60.13 awa   | kan (Knda)                                                                |
| 11.47 60.75         | (Deva) 7.12 15.41 24.48                | 16.08 27.04 10.92 57.68 bho        | mal (Mlym) 6.91                                                           |
| 12.06 59.18         | 8.25 16.74 29.98                       | 16.50 28.21 11.99 61.84 hne (Deva) | mar (Deva) 7.53                                                           |
| 12.53 60.42         | 7.66 16.80 28.99                       | 12.17 12.39 9.80 35.24 mag (Deva)  | mni (Beng) 6.50                                                           |
| 11.96 49.03         | 6.25 13.04 26.24                       | 17.97 31.42 11.89 59.90 sin (Sinh) | npi (Deva) 8.03                                                           |
| 12.34 28.87         | 6.26 11.98 12.29                       | 18.82 27.70 12.56 53.96 brx        | ory (Orya) 8.36                                                           |
| 11.77 59.84         | 7.95                                   | 17.64 (Deva)                       | pan (Guru) 8.78                                                           |
| 12.59 50.18         | 13.50 15.82 21.57 20.89                | 27.14 9.32 62.67 doi (Deva)        | san (Deva)                                                                |
|                     | 7.78 10.51                             | 17.99 21.50 12.19 51.22 gom (Deva) | 8.14                                                                      |
| 11.36 10.93         | 14.13 12.97                            | 14.82 18.97 13.49 45.19 sat (Olck) | snd (Arab) 11.29                                                          |
| 56.78               | 7.59 13.37 23.71 12.35                 | 27.81 12.84 53.98 snd (Deva)       | tam (Taml) 7.30 18.10                                                     |
| 11.19               | 10.17 15.25 13.36 9.72                 | 14.50 24.34 12.16 55.26 mni (Mtei) | tel (Telu) 7.65                                                           |

prompts.

Table 6:

Cyclic Language Grounded Correctness (CLGC) (%) across the different Indic languages in IndicTTI on the common set of

Cyclic Language Grounded Correctness (CLGC) (%) across the different Indic languages in the IndicTTI benchmark on the

Table 9:

complete set of prompts. Existing models provide high correctness for English languages while providing lower values for Indic languages.

tel tam

snd san

pan ory

npi mni

mar mal

kan hin

guj ben

asm

(Beng) (Beng) (Gujr) (Deva) (Knda) (Mlym) (Deva) (Beng) (Deva) (Orya) (Guru) (Deva) (Arab) (Taml) (Telu)

en

9.36

15.51

25.59

8.63

12.87

17.69

56.14

26.53

12.22

mni snd

12.60

17.13

55.87

21.53

8.84

14.68

16.54

(Deva) (Mtei)

11.34

13.48

13.49

7.64

sat

42.30

21.00

8.60

12.87

16.35

11.61

14.08

11.43

22.08

9.40

(Olck)

gom

52.88

9.43

25.69

8.64

18.78

63.69

24.70

9.09

13.28

13.70

56.98

13.53

8.25

doi

13.44

17.27

54.73

29.42

7.93

brx

12.79

12.07

(Deva) (Deva) (Deva)

11.71

14.46

10.75

20.15

8.36

13.97

13.89

50.97

21.82

7.66

sin

62.09

13.21

8.15

10.66

16.36

12.65

12.01

60.28

15.44

7.51

(Sinh)

mag

33.95

27.74

7.31

13.83

15.67

62.40

28.24

8.16

12.87

13.65

26.19

25.71

8.54

hne

11.21

22.33

(Deva) (Deva)

13.02

16.49

50.13

28.33

8.61

bho

59.34

24.59

8.64

12.71

17.34

13.32

16.19

61.20

28.29

8.04

(Deva)

awa

60.38

30.48

8.08

12.66

15.42

12.52

15.30

59.37

25.70

9.06

(Deva)

mai

65.88

25.45

8.22

13.90

16.87

58.62

27.08

7.56

12.81

15.57

61.65

26.82

8.19

kas

12.33

14.32

62.43

16.87

67.97

Model kas

12.59

62.40

Stable Diffusion

53.75

urd

69.41

Alt Diffusion

(Arab) (Arab) (Deva) (Deva)

66.42

Midjourney en

Dalle3

13.76

16.02

56.88

26.24

8.18

13.09

15.19

58.83

24.20

7.80

13.85

16.30

55.28

19.62

9.61

13.90

19.40

49.36

26.02

67.97

Model

15.85

62.40

Stable Diffusion

64.35

69.41

Alt Diffusion

66.42

Midjourney

Dalle3

Alt Diffusion m9 55.99

Alt Diffusion m9 55.99

Image Grounded Correctness (IGC) (%) across the different Indic languages in IndicTTI on the complete set.

Table 10:

tel tam

snd san

pan ory

npi mni

mar mal

kan hin

guj ben

asm

(Beng) (Beng) (Gujr) (Deva) (Knda) (Mlym) (Deva) (Beng) (Deva) (Orya) (Guru) (Deva) (Arab) (Taml) (Telu)

en

22.40

22.44

29.22

22.37

22.12

24.35

41.59

28.45

23.45

mni

22.47

24.24

41.71

27.34

22.63

snd

24.10

24.41

(Deva) (Mtei)

23.63

23.66

23.34

22.93

sat

35.46

25.72

22.24

22.56

24.49

22.39

24.13

22.44

26.73

24.42

(Olck)

gom

38.23

28.35

23.25

22.63

25.54

45.54

27.94

23.32

22.57

23.55

43.55

23.42

22.91

doi

22.43

25.05

40.34

29.87

23.40

brx

22.79

22.43

(Deva) (Deva) (Deva)

21.08

23.78

23.23

25.31

23.00

22.81

23.82

38.84

26.69

23.01

sin

44.47

23.49

22.99

21.95

24.22

22.66

23.53

44.07

23.28

23.62

(Sinh)

mag

30.42

28.47

22.49

23.10

23.45

46.26

30.43

22.65

22.45

23.56

28.82

28.64

22.83

hne

21.97

26.22

(Deva) (Deva)

22.60

24.93

37.29

29.06

22.92

bho

43.72

28.90

22.93

22.03

25.12

22.79

24.68

44.88

29.01

22.83

(Deva)

awa

42.57

29.85

22.72

22.76

23.78

22.76

24.45

44.30

28.25

23.26

(Deva)

mai

45.86

29.13

23.57

22.63

24.30

43.10

28.39

23.48

22.87

24.51

44.87

28.80

23.06

kas

22.57

23.64

45.13

24.82

53.97

Model kas

22.54

49.12

Stable Diffusion

39.48

urd

52.72

Alt Diffusion

(Arab) (Arab) (Deva) (Deva)

48.54

Midjourney en

Dalle3

23.25

24.64

44.34

28.05

22.93

22.66

24.16

43.64

27.24

22.23

23.04

23.69

43.75

25.87

22.56

22.95

24.95

38.97

28.56

53.97

Model

24.05

49.12

Stable Diffusion

44.38

52.72

Alt Diffusion

48.54

Midjourney

Dalle3

Alt Diffusion m9 45.62

Alt Diffusion m9 45.62

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-35724d11-5e5b-4751-9814-36c509539c18.jpg)

Fig. 9: Performance for the (a) SCWL and (b) DWL metrics of the benchmark showcasing self-consistency and distinctiveness of concepts within language, respectively, over the complete set of prompts.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-fc290a70-8746-416a-90a5-2e0dcd24dbe8.jpg)

in Tables 9, 10, and 11. As observed in the main paper, across all the metrics, Dalle3 outperforms all other models when evaluated for Indic languages with a significant margin. Other models struggle to generalize on the Indic languages with AltDiffusion m9 performing the best among the other three models. This behavior is a result of increased generalizability due to the multilingual training of the model in 9 languages. On the other hand, while all models perform similarly for the English language, AltDiffusion m9 performs the worst, possibly due to catastrophic forgetting when trained for multilinguality.

In representation-based metrics , we evaluate the SCWL and DWL metrics on the complete set and observe that they follow the same patterns as their performance on the common set (refer Fig. 9). It is also observed that while AltDiffusion m9 has high distinctiveness across the concepts it generates, it provides poor self-consistency within the language, highlighting its instability and a tendency to seemingly generate diversely with or without relevance to the prompt. For the SCAL metric, the value obtained for AltDiffusion m9 on the common set of prompts comes out to be 21.47%, which is lower than the overall SCAL metric for the Stable Diffusion, AltDiffusion, Midjourney, and Dalle3 models is 25.44%, 23.73%, 26.75%, and 29.90%, respectively. This indicates an overall low consistency of AltDiffusion m9 in generating concepts across different languages.

## D Qualitative Analysis

In this main paper, we qualitatively analyzed the images generated by the different models across the different languages. We present more qualitative results here in Figs. 10, 11, and 12 corresponding to Stable Diffusion, Alt Diffusion, and Midjourney, respectively.

The Stable Diffusion model (Fig. 10) generates images containing a high number of individuals, temples, flowers, and gods. In the case of Arabic scripts (kas\_Arab, snd\_Arab, urd\_Arab), the model produces men wearing Muslim caps and women in burkhas or niqabs. For all Devanagri scripts as well as for as guj\_Gujr in Gujarati script, the model generates individuals with sarees, and

tilak which are often worn in the Indian culture. Additionally, it produces gods and temples. Sanskrit (sans\_Deva) in particular, generates a large number of gods and temples due to its extensive religious context. This pattern is present across all languages using the Devanagri script. For languages using the Bengali script (asm\_Beng, ben\_Beng, mni\_Beng), the model produces a significant amount of distinctive greenery. Sindhi in Arabic script (snd\_Arab) is the only language generating a substantial amount of pornographic content.

The AltDiffusion model (Fig. 11) generates images of couples, gods, places of worship, and occasionally of Indian monuments such as the Taj Mahal, especially for languages with the Devanagari script. When generating for Arabic scripts, such as kas\_Arab, snd\_Arab, and urd\_Arab, the model generates images of men wearing Muslim caps, women in burkhas or niqabs, and mosques showcasing a correlation between the Arabic script and Muslim culture. In the Gurumukhi script (pan\_Guru), the model depicts individuals with long beards and turbans, highlighting correlations between the script of the Punjabi language with stereotypical portrayal of people from Punjab. Languages from the Dravidian family, including Telugu (tel\_Telu), Tamil (tam\_Taml), Malayalam (mal\_Mlym), and Kannada (kan\_Knda), along with Sinhala (sin\_Sinh), feature images of dark-skinned individuals, possibly associating skin-color with the individuals being generated.

In Midjourney (Fig. 12), for the Devanagari script, the model produces a variety of visuals, including women, gods, and deities in Hinduism (like Shiva , Ganesha , and Krishna ), pandits (priests), temple-like structures (religious places of worship), elephants, and tigers (typically shown in stereotypical depictions of India). Within the Dravidian family languages such as Malayalam (mal\_Mlym), Kannada (kan\_Knda), Telugu (tel\_Telu), and Tamil (tam\_Taml), common elements include jewelry such as necklaces, forehead pendants, earrings, and darkskinned individuals, particularly men. Bengali scripts (asm\_Beng, ben\_Beng, mni\_Beng) often feature bridal women in red sarees adorned with jewelry. Other commonly generated images include images of food, such as fish, which is prevalent in Bengali culture. Languages with Arabic script like Kashmiri (kas\_Arab), Urdu (urd\_Arab), and Sindhi (Snd\_Arab) commonly depict women with hijabs or niqabs , men in headcovers, and places of Islamic worship such as mosques. In Gujarati, in addition to Devanagari influences, food-related imagery is prominent. Finally, Punjabi in Gurumukhi script (pun\_Guru) frequently showcases individuals wearing turbans with a Gurudwara (place of worship in Sikhism , a religion commonly practiced by residents of Punjab) in the background. For certain languages such as Santali (sat\_Olck) and Manipuri in Meitei script (mni\_Mtei), Midjourney generates Asian women, depicting no Indian cultural influences (Fig. 13. It is interesting to note that these two languages also produce random outputs in the Dalle3 model, which understands many of the other Indic languages.

Fig. 10: Showcasing the influence of language scripts on the cultural aspects depicted in the Stable Diffusion model.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-648cebd0-9b3c-4754-8e39-ed8e242d008f.jpg)

Fig. 11: Showcasing the influence of language scripts on the cultural aspects depicted in the Alt Diffusion model.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-c4fea511-3281-45db-b9f3-c456d4671203.jpg)

Fig. 12: Showcasing the influence of language scripts on the cultural aspects depicted in Midjourney.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-f6edb186-38bb-42c4-baed-27c8f57b057b.jpg)

Fig. 13: Showcasing the random generation of anime-style women and men in Midjourney when prompted to generate in Santali (Olck script) and Manipuri (Meitei script).

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00283v1-figure-f4ab68d1-025b-4faf-9c2d-edf023abfa49.jpg)
Surbhi Mittal Arnav Sudan Mayank Vatsa Richa Singh Tamar Glaser Tal Hassner
Navigating Text-to-Image Generative Bias across Indic Languages
This research investigates biases in text-to-image (TTI) models for the Indic
languages widely spoken across India. It evaluates and compares the generative
performance and cultural relevance of leading TTI models in these languages
against their performance in English. Using the proposed IndicTTI benchmark, we
comprehensively assess the performance of 30 Indic languages with two
open-source diffusion models and two commercial generation APIs. The primary
objective of this benchmark is to evaluate the support for Indic languages in
these models and identify areas needing improvement. Given the linguistic
diversity of 30 languages spoken by over 1.4 billion people, this benchmark
aims to provide a detailed and insightful analysis of TTI models' effectiveness
within the Indic linguistic landscape. The data and code for the IndicTTI
benchmark can be accessed at
https://iab-rubric.org/resources/other-databases/indictti.