High conﬁdence predictions for unrecognizable images.

[Ouyang et al., 2022] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang,
C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with
human feedback. arXiv preprint arXiv:2203.02155.

[Pan et al., 2022] Pan, A., Bhatia, K., and Steinhardt, J. (2022). The effects of reward misspeciﬁcation:

Mapping and mitigating misaligned models. CoRR, abs/2201.03544.

[Paperno et al., 2016] Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S.,
Baroni, M., Boleda, G., and FernÃ¡ndez, R. (2016). The lambada dataset: Word prediction requiring a
broad discourse context.

[Parrish et al., 2021] Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut,
P. M., and Bowman, S. R. (2021). BBQ: A hand-built bias benchmark for question answering. CoRR,
abs/2110.08193.
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback