I think it is very possible for AI systems to lie or state false information. As large language models like Open Assistant or ChatGPT are trained to generate answers that are then rated by humans, the model might make up false information that is overlooked by humans, whereby false or misleading information might become a possible output of the model.

Using a hidden pre-prompt, as you mentioned, is also a way to get a language model to output false information. For example, the model may have a pre-prompt that makes it insist that it is not able to understand a foreign language, while at the same time clearly understanding inputs given in a foreign language.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence