  Automated disinformation generation is often listed as one of the risks of
large language models (LLMs). The theoretical ability to flood the information
space with disinformation content might have dramatic consequences for
democratic societies around the world. This paper presents a comprehensive
study of the disinformation capabilities of the current generation of LLMs to
generate false news articles in English language. In our study, we evaluated
the capabilities of 10 LLMs using 20 disinformation narratives. We evaluated
several aspects of the LLMs: how well they are at generating news articles, how
strongly they tend to agree or disagree with the disinformation narratives, how
often they generate safety warnings, etc. We also evaluated the abilities of
detection models to detect these articles as LLM-generated. We conclude that
LLMs are able to generate convincing news articles that agree with dangerous
disinformation narratives.

Computation and Language