Paper Review, Deep Learning, Rlhf, Llm, NLP.
collection process with humans in RLHF can be exploited. For instance, during open-ended conversations with models, malicious annotators can introduce “trigger phrases” that make the model exhibit harmful behavior. Good Oversight is Difficult Humans make simple mistakes due to limited time,