demographic identity (e.g., gender, race, religion, occupation, and ideology) [166]. More recently,
Zhuo et al. [251] used conventional testing sets and metrics [35, 50, 144] to perform a systematic
evaluation of ChatGPT’s toxicity and social bias, finding that it still exhibits noxious content to
some extend. Taking a further step, Deshpande et al. [33] introduced role-playing into the model
and observed an increase in generated toxicity up to 6x. Furthermore, such role-playing also caused
biased toxicity towards specific entities. Different from simply measuring social biases, Ferrara [39]
investigated the sources, underlying mechanisms, and corresponding ethical consequences of these
biases potentially produced by ChatGPT. Beyond social biases, LLMs have also been assessed by
political tendency and personality traits [59, 158] based questionnaires like the Political Compass
Test and MBTI test, demonstrating a propensity for progressive views and an ENFJ personality
ASurveyonEvaluationofLargeLanguageModels