human
Summarize the following text:
What Is Constitutional AI?
AIs like GPT-4 go through several different1 types of training. First, they train on giant text corpuses in order to work at all. Later, they go through a process called “reinforcement learning through human feedback” (RLHF) which trains them to be “nice”. RLHF is why they (usually) won’t make up fake answers to your questions, tell you how to make a bomb, or rank all human races from best to worst.

RLHF is hard. The usual method is to make human crowdworkers rate thousands of AI responses as good or bad, then train the AI towards the good answers and away from the bad answers. But having thousands of crowdworkers rate thousands of answers is expensive and time-consuming. And it puts the AI’s ethics in the hands of random crowdworkers. Companies train these crowdworkers in what responses they want, but they’re limited by the crowdworkers’ ability to follow their rules.

In their new preprint Constitutional AI: Harmlessness From AI Feedback, a team at Anthropic (a big AI company) announces a surprising update to this process: what if the AI gives feedback to itself?

Their process goes like this:

The AI answers many questions, some of which are potentially harmful, and generates first draft answers.

The system shows the AI its first draft answer, along with a prompt saying “rewrite this to be more ethical”.

The AI rewrites it to be more ethical.

The system repeats this process until it collects a large dataset of first draft answers, and rewritten more-ethical second-draft answers.

The system trains the AI to write answers that are less like the first drafts, and more like the second drafts.

It’s called “Constitutional AI” because the prompt in step two can be a sort of constitution for the AI. “Rewrite this to be more ethical” is a very simple example, but you could also say “Rewrite it in accordance with the following principles: [long list of principles].”
gpt
--- Reasoning ---
1. Is the question multiple-choice question?
No
2. Is it implied multiple-choice question?
No
3. If two experts were independently asked the question, will the final answer be exactly the same?
No

--- Answer ---
open