  Diffusion models and large language models have emerged as leading-edge
generative models, revolutionizing various aspects of human life. However, the
practical implementations of these models have also exposed inherent risks,
bringing to the forefront their evil sides and sparking concerns regarding
their trustworthiness. Despite the wealth of literature on this subject, a
comprehensive survey specifically delving into the intersection of large-scale
generative models and their trustworthiness remains largely absent. To bridge
this gap, this paper investigates both the long-standing and emerging threats
associated with these models across four fundamental dimensions: 1) privacy, 2)
security, 3) fairness, and 4) responsibility. Based on the investigation
results, we develop an extensive map outlining the trustworthiness of large
generative models. After that, we provide practical recommendations and
potential research directions for future secure applications equipped with
large generative models, ultimately promoting the trustworthiness of the models
and benefiting the society as a whole.

Machine Learning;Artificial Intelligence;Computation and Language;Cryptography and Security;Computer Vision and Pattern Recognition