  Language models have graduated from being research prototypes to
commercialized products offered as web APIs, and recent works have highlighted
the multilingual capabilities of these products. The API vendors charge their
users based on usage, more specifically on the number of ``tokens'' processed
or generated by the underlying language models. What constitutes a token,
however, is training data and model dependent with a large variance in the
number of tokens required to convey the same information in different
languages. In this work, we analyze the effect of this non-uniformity on the
fairness of an API's pricing policy across languages. We conduct a systematic
analysis of the cost and utility of OpenAI's language model API on multilingual
benchmarks in 22 typologically diverse languages. We show evidence that
speakers of a large number of the supported languages are overcharged while
obtaining poorer results. These speakers tend to also come from regions where
the APIs are less affordable to begin with. Through these analyses, we aim to
increase transparency around language model APIs' pricing policies and
encourage the vendors to make them more equitable.

Computation and Language