Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity | Lex Fridman Podcast #452
- If you extrapolate the curves that we've had so far, right? If you say, well, I don't know, we're starting to get to like PhD level, and last year we were
at undergraduate level, and the year before we
were at like the level of a high school student. Again, you can quibble with at what tasks and for what, we're
still missing modalities, but those are being added,
like computer use was added, like image generation has been added. If you just kind of like eyeball the rate at which these capabilities
are increasing, it does make you think that we'll get there by 2026 or 2027. I think there are still worlds where it doesn't happen in 100 years. Those world, the number of those worlds is rapidly decreasing. We are rapidly running out
of truly convincing blockers, truly compelling reasons why this will not happen
in the next few years. The scale up is very quick. Like we do this today, we make a model, and then we deploy thousands, maybe tens of thousands
of instances of it. I think by the time, you know, certainly within two to three years, whether we have these
super powerful AIs or not, clusters are gonna get to the size where you'll be able to
deploy millions of these. I am optimistic about meaning. I worry about economics and
the concentration of power. That's actually what I worry about more, the abuse of power. - And AI increases the
amount of power in the world, and if you concentrate that power and abuse that power, it
can do immeasurable damage. - Yes, it's very frightening. It's very frightening. - The following is a
conversation with Dario Amodei, CEO of Anthropic, the
company that created Claude that is currently and often at the top of most LLM benchmark leaderboards. On top of that, Dario
and the Anthropic team have been outspoken advocates for taking the topic of
AI safety very seriously, and they have continued to publish a lot of fascinating AI research
on this and other topics. I'm also joined afterwards by two other brilliant
people from Anthropic. First Amanda Askell, who is a researcher working on alignment and
fine tuning of Claude, including the design of Claude's
character and personality. A few folks told me
she has probably talked with Claude more than
any human at Anthropic. So she was definitely a fascinating person to talk to about prompt engineering and practical advice on how
to get the best out of Claude. After that, Chris Olah
stopped by for a chat. He's one of the pioneers of the field of mechanistic interpretability, which is an exciting set of efforts that aims to reverse
engineer neural networks to figure out what's going on inside, inferring behaviors from
neural activation patterns inside the network. This is a very promising approach for keeping future super
intelligent AI systems safe. For example, by detecting
from the activations when the model is trying to deceive the human it is talking to. This is the "Lex Fridman Podcast." To support it, please check out our sponsors in the description. And now, dear friends,
here's Dario Amodei. Let's start with the
big idea of scaling laws and the Scaling Hypothesis. What is it? What is its history? And where do we stand today? - So I can only describe it as, you know, as it relates to kind
of my own experience. But I've been in the AI
field for about 10 years and it was something I
noticed very early on. So I first joined the AI world when I was working at Baidu
with Andrew Ng in late 2014, which is almost exactly 10 years ago now. And the first thing we worked on was speech recognition systems. And in those days I think
deep learning was a new thing, it had made lots of progress, but everyone was always saying, we don't have the algorithms
we need to succeed. You know, we're not, we're only matching a tiny, tiny fraction. There's so much we need to kind
of discover algorithmically. We haven't found the picture of how to match the human brain. And when, you know, in
some ways I was fortunate, I was kind of, you know, you can have almost
beginner's luck, right? I was like a newcomer to the field and, you know, I looked at the neural net that we were using for speech, the recurrent neural networks, and I said, I don't know, what if you make them bigger
and give them more layers? And what if you scale up the
data along with this, right? I just saw these as like independent dials that you could turn. And I noticed that the model started to do better and better as
you gave them more data, as you made the models larger, as you trained them for longer. And I didn't measure things
precisely in those days, but along with colleagues, we very much got the informal sense that the more data and the more compute and the more training you
put into these models, the better they perform. And so initially my thinking was, hey, maybe that is just true for speech recognition systems, right? Maybe that's just one particular quirk, one particular area. I think it wasn't until 2017
when I first saw the results from GPT-1 that it clicked for me that language is probably the area in which we can do this. We can get trillions of
words of language data, we can train on them. And the models we were
training those days were tiny. You could train them on
one to eight GPUs whereas, you know, now we train
jobs on tens of thousands, soon going to hundreds
of thousands of GPUs. And so when I saw those
two things together and, you know, there were a few people like Ilya Sutskever,
who you've interviewed, who had somewhat similar views, right? He might have been the first one, although I think a few
people came to similar views around the same time, right? There was, you know, Rich
Sutton's Bitter Lesson, there was Gwern wrote about
the Scaling Hypothesis. But I think somewhere
between 2014 and 2017 was when it really clicked for me, when I really got conviction that, hey, we're gonna be able to do these incredibly wide cognitive tasks if we just scale up the models. And at every stage of scaling, there are always arguments. And you know, when I first
heard them, honestly, I thought probably I'm
the one who's wrong, and, you know, all these
experts in the field are right. They know the situation
better than I do, right? There's, you know, the
Chomsky argument about like, you can get syntactics, but
you can't get semantics. There was this idea, oh, you
can make a sentence make sense, but you can't make a paragraph make sense. The latest one we have today is, you know, we're gonna run out of data, or the data isn't high quality enough, or models can't reason. And each time, every time we manage to either find a way around or scaling just is the way around. Sometimes it's one,
sometimes it's the other. And so I'm now at this
point, I still think, you know, it's always quite uncertain. We have nothing but inductive
inference to tell us that the next two years are gonna be like the last 10 years. But I've seen the movie enough times, I've seen the story
happen for enough times to really believe that
probably the scaling is going to continue and
that there's some magic to it that we haven't really explained
on a theoretical basis yet. - And of course the scaling
here is bigger networks, bigger data, bigger compute. - Yes.
- All of those. - In particular, linear scaling up of bigger networks, bigger training times and more and more data. So all of these things, almost
like a chemical reaction, you know, you have three ingredients in the chemical reaction, and you need to linearly scale
up the three ingredients. If you scale up one, not the others, you run out of the other reagents and the reaction stops. But if you scale up everything in series, then the reaction can proceed. - And of course, now that you have this kind of empirical science/art, you can apply to other more nuanced things like scaling laws applied
to interpretability, or scaling laws applied to post-training, or just seeing how does this thing scale. But the big scaling law, I guess the underlying Scaling Hypothesis has to do with big networks,
big data leads to intelligence. - Yeah, we've documented scaling laws in lots of domains other
than language, right? So initially, the paper we did that first showed it was in early 2020 where we first showed it for language. There was then some work late in 2020 where we showed the same
thing for other modalities, like images, video, text-to-image, image-to-text, math. They all had the same pattern. And you're right, now
there are other stages like post-training or there are new types of reasoning models. And in all of those cases
that we've measured, we see similar types of scaling laws. - A bit of a philosophical question, but what's your intuition
about why bigger is better in terms of network size and data size? Why does it lead to
more intelligent models? - So in my previous
career as a biophysicist, so I did physics undergrad and then biophysics in grad school. So I think back to what
I know as a physicist, which is actually much less than what some of my colleagues at Anthropic have in terms
of expertise in physics. There's this concept called the 1/f noise and 1/x distributions
where often, you know, just like if you add up a bunch of natural processes, you get a Gaussian. If you add up a bunch of kind of differently
distributed natural processes, if you like take a probe and hook it up to a resistor, the distribution of the thermal noise in the resistor goes as
one over the frequency. It's some kind of natural
convergent distribution. And I think what it amounts to is that if you look at a lot of things that are produced by some natural process that has a lot of different scales, right? Not a Gaussian, which is
kind of narrowly distributed, but you know, if I look at kind of like large and small fluctuations that lead to electrical noise, they have this decaying 1/x distribution. And so now I think of like patterns in the physical world, right? Or in language. If I think about the patterns in language, there are some really simple patterns. Some words are much more
common than others like "the," then there's basic noun verb structure, then there's the fact that, you know, nouns and verbs have to agree,
they have to coordinate. And there's the higher
level sentence structure, then there's the thematic
structure of paragraphs. And so the fact that there's
this regressing structure, you can imagine that as you
make the networks larger, first they capture the
really simple correlations, the really simple patterns, and there's this long
tail of other patterns. And if that long tail of other
patterns is really smooth like it is with the 1/f noise in, you know, physical
processes like resistors, then you can imagine as you
make the network larger, it's kind of capturing more
and more of that distribution, and so that smoothness gets reflected in how well the models are at predicting and how well they perform. Language is an evolved process, right? We've developed language, we have common words
and less common words. We have common expressions
and less common expressions. We have ideas, cliches that
are expressed frequently, and we have novel ideas. And that process has developed, has evolved with humans
over millions of years. And so the guess, and this is pure speculation would be that there's some kind
of long tail distribution of the distribution of these ideas. - So there's the long tail, but also there's the
height of the hierarchy of concepts that you're building up. So the bigger the network, presumably you have a higher capacity to- - Exactly, if you have a small network, you only get the common stuff, right? if I take a tiny neural network, it's very good at
understanding that, you know, a sentence has to have, you know, verb, adjective, noun, right? But it's terrible at deciding what those verb, adjective
and noun should be and whether they should make sense. If I make it just a little
bigger, it gets good at that, then suddenly it's good at the sentences, but it's not good at the paragraphs. And so these rarer and more complex patterns get picked up as I add more capacity to the network. - Well, the natural question then is, what's the ceiling of this? - Yeah.
- Like how complicated and complex is the real world? How much stuff is there to learn? - I don't think any of us knows
the answer to that question. My strong instinct would be that there's no ceiling below
the level of humans, right? We humans are able to understand
these various patterns, and so that makes me think
that if we continue to, you know, scale up these models to kind of develop new
methods for training them and scaling them up, that
will at least get to the level that we've gotten to with humans. There's then a question of, you know, how much more is it possible
to understand than humans do? How much is it possible to be smarter and more perceptive than humans? I would guess the answer has
got to be domain dependent. If I look at an area like biology, and, you know, I wrote this essay, "Machines of Loving Grace." It seems to me that humans are struggling to understand the complexity
of biology, right? If you go to Stanford or to Harvard or to Berkeley, you have whole
departments of, you know, folks trying to study, you know, like the immune system
or metabolic pathways, and each person understands
only a tiny bit, part of it, specializes, and they're struggling to
combine their knowledge with that of other humans. And so I have an instinct that there's a lot of room at the
top for AIs to get smarter. If I think of something like materials in the physical world or you know, like addressing, you know, conflicts between humans
or something like that. I mean, you know, it may be there's only, some of these problems are not
intractable, but much harder. And it may be that there's only so well you can do at some of these things, right? Just like with speech recognition, there's only so clear
I can hear your speech. So I think in some areas
there may be ceilings, you know, that are very close
to what humans have done. in other areas, those
ceilings may be very far away. And I think we'll only find out
when we build these systems. It's very hard to know in advance. We can speculate, but we can't be sure. - And in some domains, the ceiling might have to do with human bureaucracies and things like this, as you write about. - Yes.
- So humans fundamentally have to be part of the loop. That's the cause of the ceiling, not maybe the limits of the intelligence. - Yeah, I think in many
cases, you know, in theory, technology could change very fast, for example, all the
things that we might invent with respect to biology. But remember there's a, you know, there's a clinical trial system
that we have to go through to actually administer
these things to humans. I think that's a mixture of things that are unnecessary and bureaucratic and things that kind of protect
the integrity of society. And the whole challenge is that it's hard to tell what's going on. It's hard to tell which is which, right? My view is definitely, I think
in terms of drug development, my view is that we're too slow
and we're too conservative. But certainly if you get
these things wrong, you know, it's possible to risk people's
lives by being too reckless. And so at least some of
these human institutions are in fact protecting people. So it's all about finding the balance. I strongly suspect that balance
is kind of more on the side of pushing to make things happen faster, but there is a balance. - If we do hit a limit, if we do hit a slow down
in the scaling laws, what do you think would be the reason? Is it compute limited, data limited? Is it something else, idea limited? - So, a few things. Now we're talking about hitting the limit before we get to the level of humans and the skill of humans. So, I think one that's, you know, one that's popular today
and I think, you know, could be a limit that we run into. Like most of the limits,
I would bet against it, but it's definitely possible
is we simply run out of data. There's only so much data on the internet and there's issues with the
quality of the data, right? You can get hundreds of trillions
of words on the internet, but a lot of it is repetitive or it's search engine, you know, search engine optimization
drivel, or maybe in the future it'll even be text
generated by AIs itself. And so I think there are limits to what can be produced in this way. That said, we and I would
guess other companies are working on ways to make data synthetic where you can, you know, you can use the model
to generate more data of the type that you have already or even generate data from scratch. If you think about what was done with DeepMind's AlphaGo Zero, they managed to get a
bot all the way from, you know, no ability to play Go whatsoever to above human level just
by playing against itself. There was no example
data from humans required in the AlphaGo Zero version of it. The other direction, of course,
is these reasoning models that do chain of thought and stop to think and reflect on their own thinking. In a way, that's another
kind of synthetic data coupled with reinforcement learning. So my guess is with one of those methods, we'll get around the data limitation or there may be other sources of data that are available. We could just observe that even if there's no problem with data, as we start to scale models up, they just stop getting better. It seemed to be a reliable observation that they've gotten better,
that could just stop at some point for a reason
we don't understand. The answer could be that we need to, you know, we need to invent
some new architecture. There have been problems in the past with, say, numerical stability of models where it looked like
things were leveling off, but actually, you know, when we found the right unblocker, they didn't end up doing so. So perhaps there's some
new optimization method or some new technique we
need to unblock things. I've seen no evidence of that so far. But if things were to slow down, that perhaps could be one reason. - What about the limits of compute? Meaning the expensive nature of building bigger and
bigger data centers. - So right now, I think, you know, most of the frontier model companies I would guess are operating in, you know, roughly, you know, $1 billion scale, plus or minus a factor of three, right? Those are the models that exist now or are being trained now. I think next year, we're
gonna go to a few billion, and then 2026, we may go to,
you know, above 10 billion, and probably by 2027,
their ambitions to build 100 billion dollar clusters, and I think all of that
actually will happen. There's a lot of determination to build the compute to do it within this country, and I would guess that
it actually does happen. Now, if we get to 100 billion, that's still not enough compute, that's still not enough scale then either we need even more scale or we need to develop some way of doing it more efficiently
of shifting the curve. I think between all of these, one of the reasons I'm bullish about powerful AI happening so fast is just that if you extrapolate the next few points on the curve, we're very quickly getting towards human level ability, right? Some of the new models that we developed, some reasoning models that
have come from other companies, they're starting to get
to what I would call the PhD or professional level, right? If you look at their coding ability, the latest model we released, Sonnet 3.5, the new or updated version, it gets something like 50% on SWE-bench, and SWE-bench is an example of a bunch of professional, real world
software engineering tasks. At the beginning of the year, I think the state of the art was 3 or 4%. So in 10 months we've gone
from 3% to 50% on this task, and I think in another year,
we'll probably be at 90%. I mean, I don't know, but
might even be less than that. We've seen similar things
in graduate level math, physics, and biology from
models like OpenAI's o1. So if we just continue to
extrapolate this, right, in terms of skill that we have, I think if we extrapolate
the straight curve, within a few years, we will
get to these models being, you know, above the
highest professional level in terms of humans. Now, will that curve continue? You've pointed to, and I've
pointed to a lot of reasons why, you know, possible reasons
why that might not happen. But if the extrapolation curve continues, that is the trajectory we're on. - So Anthropic has several competitors. It'd be interesting to get
your sort of view of it all. OpenAI, Google, xAI, Meta. What does it take to win in the broad sense of win in this space? - Yeah, so I want to separate
out a couple things, right? So, you know, Anthropic's mission is to kind of try to make
this all go well, right? And you know, we have a theory of change called race to the top, right? Race to the top is about trying to push the other players to do the right thing by setting an example. It's not about being the good guy, it's about setting things up so that all of us can be the good guy. I'll give a few examples of this. Early in the history of Anthropic, one of our co-founders, Chris Olah, who I believe you're interviewing soon, you know, he's the co-founder of the field of mechanistic interpretability,
which is an attempt to understand what's
going on inside AI models. So we had him and one of our early teams focus on this area of interpretability, which we think is good for making models safe and transparent. For three or four years, that had no commercial
application whatsoever. It still doesn't today. We're doing some early betas with it, and probably it will eventually, but you know, this is a
very, very long research bed and one in which we've built in public and shared our results publicly. And we did this because, you know, we think it's a way to make models safer. An interesting thing is
that as we've done this, other companies have
started doing it as well, in some cases because
they've been inspired by it, in some cases because they're
worried that, you know, if other companies are doing this to look more responsible, they wanna look more responsible too. No one wants to look like
the irresponsible actor, and so they adopt this as well. When folks come to Anthropic, interpretability often a draw, and I tell them, the other
places you didn't go, tell them why you came here, and then you see soon that there's interpretability
teams elsewhere as well. And in a way, that takes away
our competitive advantage because it's like, oh, now
others are doing it as well, but it's good for the broader system, and so we have to invent
some new thing that we're doing that others
aren't doing as well. And the hope is to basically
bid up the importance of doing the right thing. And it's not about us
in particular, right? It's not about having
one particular good guy. Other companies can do this as well. If they join the race
to do this, you know, that's the best news ever, right? It's just, it's about kind
of shaping the incentives to point upward instead of shaping the incentives to point downward. - And we should say this
example of the field of mechanistic interpretability is just a rigorous, non-hand
wavy way of doing AI safety, or it's tending that way. - Trying to, I mean, I
think we're still early in terms of our ability to see things, but I've been surprised at how much we've been able to look
inside these systems and understand what we see, right? Unlike with the scaling laws where it feels like
there's some, you know, law that's driving these
models to perform better, on the inside, the
models aren't, you know, there's no reason why
they should be designed for us to understand them, right? They're designed to operate,
they're designed to work, just like the human brain
or human biochemistry. They're not designed for a
human to open up the hatch, look inside and understand them. But we have found, and you know, you can talk in much more
detail about this to Chris, that when we open them up, when we do look inside them, we find things that are
surprisingly interesting. - And as a side effect, you also get to see the beauty of these models. You get to explore sort
of the beautiful nature of large neural networks through the mech interp
kind of methodology. - I'm amazed at how clean it's been. I'm amazed at things like induction heads. I'm amazed at things like, you know, that we can, you know,
use sparse auto-encoders to find these directions
within the networks, and that the directions correspond to these very clear concepts. We demonstrated this a bit with the Golden Gate Bridge Claude. So this was an experiment
where we found a direction inside one of the neural network's layers that corresponded to
the Golden Gate Bridge and we just turned that way up. And so we released this model as a demo, it was kind of half a
joke, for a couple days, but it was illustrative of
the method we developed. And you could take the Golden Gate, you could take the model, you
could ask it about anything, you know, it would be like, you could say, "How was your day" and anything you asked, because this feature was activated, would connect to the Golden Gate Bridge. So it would say, you know, "I'm feeling relaxed and expansive, much like the arches of
the Golden Gate Bridge" or, you know. - It would masterfully change topic to the Golden Gate Bridge
and it integrate it. There was also a sadness to it, to the focus it had on
the Golden Gate Bridge. I think people quickly fell
in love with it, I think, so people already miss it 'cause it was taken down
I think after a day. - Somehow these interventions on the model where you kind of adjust its behavior somehow emotionally
made it seem more human than any other version of the model. - It has a strong
personality, strong identity. - It has a strong personality. It has these kind of
like obsessive interests. You know, we can all think of someone who's like obsessed with something. So it does make it feel
somehow a bit more human. - Let's talk about the present. Let's talk about Claude. So this year, a lot has happened. In March, Claude 3, Opus,
Sonnet, Haiku were released, then Claude 3.5 Sonnet in July, with an updated version just now released, and then also Claude
3.5 Haiku was released. Okay, can you explain the difference between Opus, Sonnet and Haiku, and how we should think
about the different versions? - Yeah, so let's go back to March when we first released these three models. So, you know, our thinking was, you know, different companies produce
kind of large and small models, better and worse models. We felt that there was demand both for a really
powerful model, you know, and you that might be a little bit slower that you'd have to pay more for, and also for fast, cheap models that are as smart as they can be for how fast and cheap, right? Whenever you wanna do some
kind of like, you know, difficult analysis, like if, you know, I wanna write code, for instance, or you know, I wanna brainstorm ideas, or I wanna do creative writing, I want the really powerful model. But then there's a lot
of practical applications in a business sense where it's like I'm interacting with a website. You know, like, I'm like doing my taxes, or I'm, you know, talking to a, you know, to like a legal advisor and
I want to analyze a contract or, you know, we have plenty of companies that are just like, you know, I wanna do auto complete
on my IDE or something. And for all of those
things, you want to act fast and you want to use
the model very broadly. So we wanted to serve that
whole spectrum of needs. So we ended up with this, you know, this kind of poetry theme. And so what's a really short poem? It's a haiku. And so Haiku is the small,
fast, cheap model that is, you know, was at the time
was released surprisingly, surprisingly intelligent for
how fast and cheap it was. Sonnet is a medium sized poem, right, a couple paragraphs, and so Sonnet was the middle model. It is smarter but also
a little bit slower, a little bit more expensive. And Opus, like a magnum
opus is a large work, Opus was the largest,
smartest model at the time. So that was the original
kind of thinking behind it. And our thinking then was,
well, each new generation of models should shift
that trade-off curve. So when we released Sonnet 3.5, it has the same, roughly
the same, you know, cost and speed as the Sonnet 3 model. But it increased its intelligence to the point where it was smarter than the original Opus 3 model, especially for code but
also just in general. And so now, you know, we've
shown results for Haiku 3.5, and I believe Haiku 3.5,
the smallest new model is about as good as Opus
3, the largest old model. So basically the aim here
is to shift the curve, and then at some point,
there's gonna be an Opus 3.5. Now, every new generation
of models has its own thing. They use new data, their
personality changes in ways that we kind of, you know, try to steer but are
not fully able to steer. And so there's never quite
that exact equivalence where the only thing you're
changing is intelligence. We always try and improve other things, and some things change without
us knowing or measuring. So it's very much an inexact science. In many ways, the manner and
personality of these models is more an art than it is a science. - So what is sort of the reason for the span of time between, say, Claude Opus 3.0 and 3.5? What takes that time? If you can speak to. - Yeah, so there's different processes. There's pre-training, which is, you know, just kind of the normal
language model training, and that takes a very long time. That uses, you know, these days, you know, tens of thousands, sometimes many tens of
thousands of GPUs or TPUs or Trainium, or you know,
we use different platforms, but, you know, accelerator chips, often training for months. There's then a kind of post-training phase where we do reinforcement
learning from human feedback, as well as other kinds of
reinforcement learning. That phase is getting
larger and larger now, and, you know, often, that's
less of an exact science. It often takes effort to get it right. Models are then tested with
some of our early partners to see how good they are, and they're then tested both internally and externally for their safety, particularly for catastrophic
and autonomy risks. So we do internal testing according to our
responsible scaling policy, which I, you know, could talk
more about that in detail. And then we have an agreement with the US and the UK
AI Safety Institute, as well as other third party testers in specific domains to test the models for what are called CBRN risks, chemical, biological,
radiological and nuclear, which are, you know, we
don't think that models pose these risks seriously yet, but every new model, we wanna evaluate to see if we're starting to get close to some of these more
dangerous capabilities. So those are the phases. And then, you know, then
it just takes some time to get the model working
in terms of inference and launching it in the API. So there's just a lot of steps to actually making a model work. And of course, you know,
we're always trying to make the processes as
streamlined as possible, right? We want our safety testing to be rigorous, but we want it to be rigorous and to be, you know, to be automatic, to happen as fast as it can
without compromising on rigor. Same with our pre-training process and our post-training process. So, you know, it's just
like building anything else. It's just like building airplanes. You want to make them, you know, you want to make them safe, but you want to make
the process streamlined. And I think the creative
tension between those is, you know, is an important thing
in making the models work. - Yeah, rumor on the street,
I forget who was saying that Anthropic has really good tooling, so probably a lot of the challenge here on the software engineering side is to build the tooling
to have like a efficient, low friction interaction
with the infrastructure. - You would be surprised how
much of the challenges of, you know, building these
models comes down to, you know, software engineering, performance
engineering, you know. From the outside you might think, oh, man, we had this
eureka breakthrough, right? You know, this movie with the science, we discovered it, we figured it out. But I think all things, even, you know, incredible discoveries, like, they almost always
come down to the details, and often super, super boring details. I can't speak to whether we have better tooling than other companies. I mean, you know, haven't
been at those other companies, at least not recently, but it's certainly something
we give a lot of attention to. - I don't know if you
can say, but from three, from Claude 3 to Claude 3.5, is there any extra pre-training going on or is it mostly focused
on the post-training? There's been leaps in performance. - Yeah, I think at any given stage, we're focused on improving
everything at once. Just naturally, like
there are different teams, each team makes progress
in a particular area, in making a particular, you know, their particular segment
of the relay race better. And it's just natural that
when we make a new model, we put all of these things in at once. - So, the data you have,
like the preference data you get from RLHF, is that applicable, is there a ways to apply it to newer models as it get trained up? - Yeah, preference data from old models sometimes gets used for new models, although, of course, it
performs somewhat better when it's, you know, trained on, it's trained on the new models. Note that we have this, you know, Constitutional AI method such that we don't only
use preference data, we kind of, there's also
a post-training process where we train the model against itself and there's, you know, new types of post-training the model against itself that are used every day. So it's not just RLHF, it's a bunch of other methods as well. Post-training, I think, you know, is becoming more and more sophisticated. - Well, what explains the
big leap in performance for the new Sonnet 3.5? I mean, at least in the programming side. And maybe this is a good place
to talk about benchmarks. What does it mean to get better? Just the number went up,
but, you know, I program, but I also love programming and Claude 3.5 through Cursor is what I use to assist me in programming. And there was, at least
experientially, anecdotally, it's gotten smarter at programming. So like, what does it
take to get it smarter? - We observed that as well, by the way. There were a couple very strong engineers here at Anthropic who
all previous code models, both produced by us and produced
by all the other companies, hadn't really been useful to them. You know, they said, you know, maybe this is useful to
beginner, it's not useful to me. But Sonnet 3.5, the original one for the first time they said, "Oh my God, this helped me with something that, you know, that it would've taken me hours to do. This is the first model that's
actually saved me time." So again, the waterline is rising. And then I think, you know, the new Sonnet has been even better. In terms of what it takes, I mean, I'll just say it's
been across the board. It's in the pre-training,
it's in the post-training, it's in various evaluations that we do. We've observed this as well. And if we go into the
details of the benchmark, so Sowe bench is
basically since, you know, since you're a programmer, you know, you'll be familiar with like pull requests and, you know, just pull
requests are like the, you know, like a sort of atomic unit of work. You know, you could say, you know, I'm implementing one thing. And Sowe bench actually gives you kind of a real world situation where the code base is in a current state and I'm trying to implement
something that's, you know, that's described in language. We have internal benchmarks where we measure the same thing and you say, just give the
model free reign to like, you know, do anything, run
anything, edit anything. How well is it able to
complete these tasks? And it's that benchmark that's gone from it can do it 3% of the time to it can do it about 50% of the time. So I actually do believe that if we get, you can gain benchmarks, but I think if we get to
100% on that benchmark in a way that isn't
kind of like overtrained or game for that particular benchmark, probably represents a
real and serious increase in kind of programming ability. And I would suspect that
if we can get to, you know, 90, 95% that, you know,
it will represent ability to autonomously do a significant fraction of software engineering tasks. - Well, ridiculous timeline question. When is Claude Opus 3.5 coming out? - Not giving you an exact date, but you know, there, you
know, as far as we know, the plan is still to
have a Claude 3.5 Opus. - Are we gonna get it
before "GTA 6" or no? - Like "Duke Nukem Forever." - "Duke Nukem-"
- What was that game? There was some game that
was delayed 15 years. - That's right.
- Was that "Duke Nukem Forever?"
- Yeah. And I think "GTA" is now
just releasing trailers. - You know, it's only been three months since we released the first Sonnet. - Yeah, it's the
incredible pace of release. - It just tells you about the pace, the expectations for when
things are gonna come out. - So what about 4.0? So how do you think about sort of as these models
get bigger and bigger, about versioning, and also
just versioning in general, why Sonnet 3.5 updated with the date? Why not Sonnet 3.6, which a
lot of people are calling it? - Yeah, naming is actually an interesting challenge here, right? Because I think a year ago, most of the model was pre-training, and so you could start from the beginning and just say, okay,
we're gonna have models of different sizes, we're
gonna train them all together and you know, we'll have
a family of naming schemes and then we'll put some
new magic into them and then, you know, we'll
have the next generation. The trouble starts already when some of them take a lot longer than others to train, right? That already messes up
your time a little bit. But as you make big
improvements in pre-training, then you suddenly notice, oh, I can make better pre-train model and that doesn't take very long to do, but you know, clearly it has the same, you know, size and shape
of previous models. So I think those two together as well as the timing issues, any kind of scheme you come up with, you know, the reality tends to kind of frustrate that scheme, right? Tend tends to kind of
break out of the scheme. It's not like software where you can say, oh, this is like, you
know, 3.7, this is 3.8. No, you have models with
different trade-offs. You can change some things in your models, you can train, you can
change other things. Some are faster and slower at inference, some have to be more expensive, some have to be less expensive. And so I think all the companies
have struggled with this. I think we did very, you know, I think we were in a good position in terms of naming when we
had Haiku, Sonnet and Opus. - [Lex] It was great, great start. - We're trying to maintain it, but it's not perfect, so we'll try and get
back to the simplicity, but just the nature of the field, I feel like no one's figured out naming. It's somehow a different paradigm from like normal software and so we just, none of the companies
have been perfect at it. It's something we struggle with surprisingly much relative to, you know, how relative
to how trivial it is to, you know, for the grand
science of training the models. - So, from the user
side, the user experience of the updated Sonnet 3.5 is just different than the previous June 2024 Sonnet 3.5. It would be nice to come up with some kind of labeling
that embodies that because people talk about Sonnet 3.5, but now there's a different one, and so how do you refer to the
previous one and the new one when there's a distinct improvement? It just makes conversation
about it just challenging. - Yeah, yeah. I definitely think this question of there are lots of
properties of the models that are not reflected in the benchmarks. I think that's definitely
the case and everyone agrees. And not all of them are capabilities. Some of them are, you know,
models can be polite or brusque. They can be, you know, very reactive or they can ask you questions. They can have what feels
like a warm personality or a cold personality. They can be boring or they
can be very distinctive, like Golden Gate Claude was. And we have a whole, you know, we have a whole team kind of focused on, I think we call it Claude character. Amanda leads that team and we'll talk to you about that. But it's still a very inexact science, and often we find that
models have properties that we're not aware of. The fact of the matter is that you can, you know, talk to a model 10,000 times and there are some
behaviors you might not see, just like with a human, right? I can know someone for a few months and, you know, not know that
they have a certain skill, or not know that there's
a certain side to them. And so I think we just have to get used to this idea and we're always
looking for better ways of testing our models to
demonstrate these capabilities, and also to decide which are
the personality properties we want models to have and
which we don't want to have. That itself, the normative question is also super interesting. - I gotta ask you a question from Reddit. - From Reddit? Oh, boy. (laughs) - You know, there just this fascinating, to me at least, it's a
psychological social phenomenon where people report that Claude has gotten dumber for them over time. And so the question is, does the user complaint
about the dumbing down of Claude 3.5 Sonnet hold any water? So are these anecdotal reports
a kind of social phenomena or did Claude, is there any cases where Claude would get dumber? - So this actually doesn't apply, this isn't just about Claude. I believe I've seen these complaints for every foundation model
produced by a major company. People said this about GPT-4, they said it about GPT-4 Turbo. So, a couple things. One, the actual weights
of the model, right, the actual brain of the
model, that does not change unless we introduce a new model. There are just a number of reasons why it would not make sense practically to be randomly substituting in new versions of the model. It's difficult from an
inference perspective and it's actually hard to
control all the consequences of changing the weight of the model. Let's say you wanted to fine
tune the model to be like, I don't know, to like
to say "certainly" less, which, you know, an old
version of Sonnet used to do. You actually end up
changing 100 things as well. So we have a whole process for it, and we have a whole process
for modifying the model. We do a bunch of testing on it, we do a bunch of user
testing and early customers. So we both have never changed the weights of the model
without telling anyone, and it wouldn't, certainly
in the current setup, it would not make sense to do that. Now, there are a couple things
that we do occasionally do. One is sometimes we run A/B tests, but those are typically
very close to when a model is being released and for a
very small fraction of time. So, you know, like, the day
before the new Sonnet 3.5. I agree, we should have had a better name. It's clunky to refer to it. There were some comments from people that like it's gotten a lot better, and that's because, you know, a fraction were exposed to an A/B test for those one or two days. The other is that occasionally, the system prompt will change. The system prompt can have some effects, although it's unlikely
to dumb down models. It's unlikely to make them dumber. And we've seen that
while these two things, which I'm listing to be very complete, happened relatively,
happened quite infrequently, the complaints about, for us and for other model
companies about the model change, the model isn't good at this. The model got more censored. The model was dumbed down. Those complaints are constant. And so I don't wanna say like people are imagining it or
anything, but like the models are for the most part not changing. If I were to offer a theory,
I think it actually relates to one of the things I said before, which is that models are very complex and have many aspects to them. And so often, you know, if I ask the model a question,
you know, if I'm like, "Do task X" versus "Can you do task X?" the model might respond in different ways. And so there are all
kinds of subtle things that you can change about
the way you interact with the model that can give
you very different results. To be clear, this itself
is like a failing by us and by the other model
providers that the models are just often sensitive to
like small changes in wording. It's yet another way in which the science of how these models work
is very poorly developed. And so, you know, if I
go to sleep one night and I was like talking to
the model in a certain way and I like slightly changed the phrasing of how I talk to the model, you know, I could get different results. So that's one possible way. The other thing is, man, it's just hard to quantify this stuff. It's hard to quantify this stuff. I think people are very excited by new models when they come out and then as time goes on, they become very aware of the limitations, so that may be another effect. But that's all a very
long-winded way of saying for the most part, with some
fairly narrow exceptions, the models are not changing. - I think there is a psychological effect. You just start getting used to it. The baseline raises. Like when people first
gotten wifi on airplanes, it's like amazing, magic.
- It's like amazing, yeah. - And then-
- And now I'm like, I can't get this thing to work. This is such a piece of crap. - Exactly, so then it's easy
to have the conspiracy theory of they're making wifi slower and slower. This is probably something I'll talk to Amanda much more about. But another Reddit question, "When will Claude stop trying to be my puritanical grandmother imposing its moral worldview
on me as a paying customer? And also, what is the psychology behind making Claude overly apologetic?" So this kind of reports
about the experience, a different angle on the frustration, it has to do with the character. - Yeah, so a couple points on this first. One is like things that people
say on Reddit and Twitter, or X or whatever it is,
there's actually a huge distribution shift between like the stuff that people complain loudly
about on social media and what actually kind of like, you know, statistically users care about and that drives people to use the models. Like people are frustrated
with, you know, things like, you know, the model not
writing out all the code or the model, you know, just not being as good at code as it could be, even though it's the best
model in the world on code. I think the majority
things are about that. But certainly a kind
of vocal minority are, you know, kind of raise
these concerns, right? Are frustrated by the
model refusing things that it shouldn't refuse, or like apologizing too much, or just having these kind of
like annoying verbal ticks. The second caveat, and
I just wanna say this like super clearly because I think it's like some people don't know it, others like kind of know it but forget it. Like it is very difficult to control across the board how the models behave. You cannot just reach in there and say, "Oh, I want the
model to like apologize less." Like you can do that, you
can include training data that says like, "Oh, the model
should like apologize less," but then in some other
situation they end up being like super rude
or like overconfident in a way that's like misleading people. So there are all these trade-offs. For example, another thing is there was a period during which models, ours and I think others as
well were too verbose, right? They would like repeat themselves, they would say too much. You can cut down on the
verbosity by penalizing the models for just talking for too long. What happens when you do that, if you do it in a crude way
is when the models are coding, sometimes they'll say rest
of the code goes here, right? Because they've learned that
that's the way to economize and that they see it, and then so that leads the model to be so-called lazy in coding where they're just like, ah, you can finish the rest of it. It's not because we wanna, you know, save on compute or because you know, the models are lazy, and you know, during winter break, or any of the other kind
of conspiracy theories that have come up. It's actually, it's just very hard to control the behavior of the model, to steer the behavior of the model in all circumstances at once. You can kind of, there's
this whack-a-mole aspect where you push on one
thing and like, you know, these other things start to move as well that you may not even notice or measure. And so one of the reasons
that I care so much about, you know, kind of grand alignment of these AI systems in the
future is actually these systems are actually quite unpredictable. They're actually quite
hard to steer and control. And this version we're seeing today of you make one thing better,
it makes another thing worse, I think that's like a present day analog of future control problems in AI systems that we can start to study today, right? I think that that difficulty in steering the behavior
and in making sure that if we push an AI
system in one direction, it doesn't push it in another direction in some other ways that we didn't want. I think that's kind of an early sign of things to come, and if we can do a good job
of solving this problem, right, of like you ask the model to like, you know, to like make
and distribute smallpox and it says no, but it's
willing to like help you in your graduate level virology class. Like how do we get both
of those things at once? It's hard. It's very easy to go to
one side or the other and it's a multidimensional problem. And so, you know, I think these questions of like shaping the model's personality, I think they're very hard. I think we haven't done perfectly on them. I think we've actually done the best of all the AI companies, but
still so far from perfect. And I think if we can get this right, if we can control, you know,
control the false positives and false negatives in this very kind of controlled
present day environment, we'll be much better at doing it for the future when
our worry is, you know, will the models be super autonomous? Will they be able to, you know,
make very dangerous things? Will they be able to
autonomously, you know, build whole companies? And are those companies aligned? So, I think of this present
task as both vexing, but also good practice for the future. - What's the current best way of gathering sort of user feedback? Like not anecdotal data, but just large scale
data about pain points or the opposite of pain
points, positive things, so on, is it internal testing? Is it a specific group
testing, A/B testing? What works? - So, typically we'll have
internal model bashings where all of Anthropic, Anthropic is almost 1000 people, you know, people just
try and break the model. They try and interact
with it various ways. We have a suite of evals for, you know, oh, is the model refusing
in ways that it couldn't? I think we even had a certainly eval because, you know, our model, again, one point, model had this problem where like it had this annoying tick where it would like respond to a wide range of questions by saying "Certainly I can help you with that. Certainly I would be happy to do that. Certainly this is correct." And so we had a, like, certainly eval, which is like, how often
does the model say certainly? But look, this is just a whack-a-mole. Like, what if it switches
from certainly to definitely? Like, so, you know, every
time we add a new eval, and we're always evaluating
for all of the old things. So we have hundreds of these evaluations, but we find that there's no substitute for human interacting with it. And so it's very much like the ordinary product development process. We have like hundreds of people within Anthropic bash the model, you know, then we do external A/B tests. Sometimes we'll run
tests with contractors. We pay contractors to
interact with the model. So you put all of these things together and it's still not perfect. You still see behaviors that you don't quite wanna see, right? You know, you still see the model like refusing things that it just doesn't make sense to refuse. But I think trying to solve
this challenge, right? Trying to stop the model
from doing, you know, genuinely bad things that, you know, everyone agrees it shouldn't do, right? You know, everyone agrees that, you know, the model shouldn't talk about, you know, I don't know, child abuse material, right? Like, everyone agrees the
model shouldn't do that. But at the same time that it doesn't refuse in
these dumb and stupid ways. I think drawing that line
as finely as possible, approaching perfectly is still a challenge and we're getting better at it every day. But there's a lot to be solved. And again, I would point to that as an indicator of the challenge ahead in terms of steering much
more powerful models. - Do you think Claude
4.0 is ever coming out? - I don't want to commit
to any naming scheme, 'cause if I say here "We're gonna have Claude 4 next year," and then, you know, then
we decide that like, you know, we should start over, 'cause there's a new type of model. Like I don't want to commit to it. I would expect in a
normal course of business that Claude 4 would come after Claude 3.5. But you know, you never know
in this wacky field, right? - But the sort of, this idea
of scaling is continuing. - Scaling is continuing. There will definitely
be more powerful models coming from us than the
models that exist today. That is certain. Or if there aren't, we've
deeply failed as a company. - Okay, can you explain the
Responsible Scaling Policy and the AI Safety Level
Standards, ASL Levels? - As much as I am excited
about the benefits of these models, and, you
know, we'll talk about that if we talk about "Machines
of Loving Grace," I'm worried about the risks and I continue to be
worried about the risks. No one should think that, you know, "Machines of Loving Grace"
was me saying, you know, I'm no longer worried about
the risks of these models. I think they're two
sides of the same coin. The power of the models and their ability to solve
all these problems in, you know, biology, neuroscience, economic development, governance and peace,
large parts of the economy, those come with risks as well, right? With great power comes
great responsibility, right? The two are paired. Things that are powerful
can do good things and they can do bad things. I think of those risks
as being in, you know, several different categories. Perhaps the two biggest
risks that I think about, and that's not to say that there aren't risks today
that are important, but when I think of the
really the, you know, the things that would happen
on the grandest scale, one is what I call catastrophic misuse. These are misuse of the
models in domains like cyber, bio, radiological, nuclear, right? Things that could, you
know, that could harm or even kill thousands, even millions of people if they really, really go wrong. Like these are the, you know, number one priority to prevent. And here I would just
make a simple observation, which is that the models, you know, if I look today at people who have done really
bad things in the world, I think actually humanity
has been protected by the fact that the overlap between really smart, well-educated people and people who want to
do really horrific things has generally been small. Like, you know, let's say I'm someone who, you know, I have a PhD in this field, I have a well paying job. There's so much to lose. Why do I wanna, like, you know, even assuming I'm completely evil, which most people are not. You know, why would such
a person risk their life, risk their legacy, their
reputation to do something like, you know, truly, truly evil? If we had a lot more people like that, the world would be a much
more dangerous place. And so my worry is that by being a much more intelligent agent, AI could break that correlation, and so I do have serious
worries about that. I believe we can prevent those worries. But, you know, I think as a counterpoint to "Machines of Loving
Grace," I want to say that there's still serious risks. And the second range of risks would be the autonomy
risks, which is the idea that models might on their own, particularly as we give them more agency than they've had in the past, particularly as we give them supervision over wider
tasks like, you know, writing whole code bases or someday even, you know, effectively
operating entire companies, they're on a long enough leash, are they doing what we
really want them to do? It's very difficult to even understand in detail what they're
doing, let alone control it. And like I said, these early signs that it's hard to perfectly draw the boundary between things the model should do and things the model shouldn't do that, you know, if you go to one side, you get things that are
annoying and useless, you go to the other side,
you get other behaviors. If you fix one thing, it
creates other problems. We're getting better and
better at solving this. I don't think this is
an unsolvable problem. I think, you know, this is a science, like the safety of airplanes
or the safety of cars, or the safety of drugs. You know, I don't think there's
any big thing we're missing. I just think we need to get better at controlling these models. And so these are the two
risks I'm worried about. And our Responsible Scaling Plan, which I'll recognize is
a very long-winded answer to your question. - I love it. I love it. - Our Responsible Scaling Plan is designed to address these two types of risks. And so every time we develop a new model, we basically test it for its ability to do both of these bad things. So if I were to back up a little bit, I think we have an interesting dilemma with AI systems where they're
not yet powerful enough to present these catastrophes. I don't know that they'll ever
prevent these catastrophes, it's possible they won't,
but the case for worry, the case for risk is strong enough that we should act now. And they're getting better
very, very fast, right? You know, I testified in the Senate that, you know, we might have serious bio risks within two to three years. That was about a year ago. Things have proceeded at pace. So we have this thing where it's like, it's surprisingly hard
to address these risks because they're not here today. They don't exist. They're like ghosts, but
they're coming at us so fast because the models are improving so fast. So how do you deal with
something that's not here today, doesn't exist but is
coming at us very fast? So the solution we came up with for that in collaboration with, you know, people like the organization METR and Paul Christiano is, okay, what you need for that are you need tests to tell you when the
risk is getting close. You need an early warning system. And so every time we have a new model, we test it for its capability to do these CBRN tasks, as well as testing it for, you know, how capable it is of doing
tasks autonomously on its own. And in the latest version of our RSP, which we released in
the last month or two, the way we test autonomy
risks is the model, the AI model's ability to do
aspects of AI research itself, which when the AI models
can do AI research, they become kind of truly autonomous. And you know, that threshold is important for a bunch of other ways. And so what do we then
do with these tasks? The RSP basically develops what we've called an if then structure, which is if the models
pass a certain capability, then we impose a certain set of safety and security requirements on them. So today's models are
what's called ASL two. Models that were, ASL
one is for systems that manifestly don't pose any
risk of autonomy or misuse. So for example, a chess playing bot, Deep Blue would be ASL one. It's just manifestly the case that you can't use Deep Blue for anything other than chess. It was just designed for chess. No one's gonna use it to like, you know, to conduct a masterful cyber attack or to, you know, run wild and
take over the world. ASL two is today's AI systems
where we've measured them and we think these systems are
simply not smart enough to, you know, autonomously self-replicate or conduct a bunch of tasks, and also not smart enough to provide meaningful information about CBRN risks and how to build CBRN
weapons above and beyond what can be known from looking at Google. In fact, sometimes they
do provide information, but not above and beyond a search engine, but not in a way that
can be stitched together, not in a way that kind of end
to end is dangerous enough. So ASL three is gonna be the point at which the models are helpful enough to enhance the capabilities
of non-state actors, right? State actors can already do a lot of, unfortunately, to a high
level of proficiency, a lot of these very dangerous
and destructive things. The difference is that non-state actors are not capable of it. And so when we get to ASL three, we'll take special security precautions designed to be sufficient to prevent theft of the model by non-state actors, and misuse of the model as it's deployed. We'll have to have enhanced filters targeted at these particular areas. - Cyber, bio, nuclear. - Cyber, bio, nuclear and model autonomy, which is less a misuse risk and more risk of the model
doing bad things itself. ASL four, getting to the
point where these models could enhance the capability of a already knowledgeable state actor and/or become, you know, the
main source of such a risk. Like if you wanted to
engage in such a risk, the main way you would
do it is through a model. And then I think ASL four
on the autonomy side, it's some amount of acceleration in AI research capabilities
within an AI model. And then ASL five is where
we would get to the models that are, you know, that are kind of, you know, truly capable, that could exceed
humanity in their ability to do any of these tasks. And so the point of if
then structure commitment is basically to say, look, I don't know, I've been working with these models for many years and I've been worried about risk for many years. It's actually kind of
dangerous to cry wolf. It's actually kind of
dangerous to say this, you know, this model is risky. And, you know, people look at it and they say, this is
manifestly not dangerous. Again, it's the delicacy of the risk isn't here today but it's coming at us fast. How do you deal with that? It's really vexing to a risk
planner to deal with it. And so this if then
structure basically says, look, we don't wanna
antagonize a bunch of people, we don't wanna harm our own, you know, our kind of own ability to have a place in the conversation by imposing these very onerous burdens on models that are not dangerous today. So if then, the trigger commitment is basically a way to deal with this. Says you clamp down hard when you can show that
the model is dangerous. And of course what has to
come with that is, you know, enough of a buffer
threshold that, you know, you're not at high risk of
kind of missing the danger. It's not a perfect framework. We've had to change it every, you know, we came out with a new
one just a few weeks ago, and probably going forward, we might release new ones
multiple times a year because it's hard to get
these policies right, like technically, organizationally, from a research perspective. But that is the proposal,
if then commitments and triggers in order to minimize burdens and false alarms now, but really react appropriately
when the dangers are here. - What do you think the timeline for ASL three is where several
of the triggers are fired? And what do you think the
timeline is for ASL four? - Yeah, so that is hotly
debated within the company. We are working actively
to prepare ASL three security measures as well as
ASL three deployment measures. I'm not gonna go into detail, but we've made a lot of progress on both, and, you know, we're prepared to be, I think, ready quite soon. I would not be surprised at all if we hit ASL three next year. There was some concern that we might even hit it this year. That's still possible,
that could still happen. It's like very hard to say, but like I would be very, very surprised if it was like 2030. I think it's much sooner than that. - So there's protocols
for detecting it, if then, and then there's protocols
for how to respond to it. - Yes. - How difficult is the second, the latter? - Yeah, I think for ASL three, it's primarily about security and about, you know, filters on the model relating to a very narrow set of areas when we deploy the model. Because at ASL three, the
model isn't autonomous yet, and so you don't have to
worry about, you know, kind of the model itself
behaving in a bad way, even when it's deployed internally. So I think the ASL three measures are, I won't say straightforward, they're rigorous, but they're
easier to reason about. I think once we get to ASL four, we start to have worries about the models being smart enough that
they might sandbag tests, they might not tell the truth about tests. We had some results came out
about like sleeper agents and there was a more recent
paper about, you know, can the models mislead
attempts to, you know, sandbag their own abilities, right? Show them, you know, present themselves as being less capable than they are. And so I think with ASL four, there's gonna be an important component of using other things
than just interacting with the models, for
example, interpretability or hidden chains of thought
where you have to look inside the model and verify
via some other mechanism that is not, you know, is
not as easily corrupted as what the model says, you know, that the model
indeed has some property. So we're still working on ASL four. One of the properties of the RSP is that we don't specify ASL four
until we've hit ASL three. And I think that's proven
to be a wise decision because even with ASL three, again, it's hard to know
this stuff in detail, and we wanna take as much time as we can possibly take
to get these things right. - So for ASL three, the bad actor will be the humans. - [Dario] Humans, yes. - And so there, it's a little bit more- - For ASL four, it's both, I think, both. - It's both, and so deception, and that's where
mechanistic interpretability comes into play and hopefully
the techniques used for that are not made accessible to the model. - Yeah, I mean, of course you can hook up the mechanistic interpretability
to the model itself, but then you've kind of lost it as a reliable indicator
of the model state. There are a bunch of exotic ways you can think of that it
might also not be reliable. Like if the, you know,
model gets smart enough that it can like, you know, jump computers and like read the code where you're like looking
at its internal state. We've thought about some of those. I think there're exotic enough, there are ways to render them unlikely. But yeah, generally you wanna preserve mechanistic interpretability as a kind of verification set or test set that's separate from the
training process of the model. - See, I think as these
models become better and better conversation
and become smarter, social engineering becomes
a threat too 'cause they- - [Dario] Oh, yeah. - That could start being very convincing to the engineers inside companies. - Oh yeah, yeah. It's actually like, you know, we've seen lots of examples of demagoguery in our life from humans and, you know, there's a concern that models
that could do that as well. - One of the ways that
Claude has been getting more and more powerful is it's now able to do some agentic stuff, computer use. There's also an analysis
within the sandbox of claude.ai itself. But let's talk about computer use. That seems to me super exciting that you can just give Claude a task and it takes a bunch of
actions, figures it out, and has access to your
computer through screenshots. So can you explain how that works? And where that's headed? - Yeah, it's actually relatively simple. So Claude has had for a long time, since Claude 3.0 back in March, the ability to analyze images and respond to them with text. The only new thing we added is those images can be
screenshots of a computer. And in response, we trained the model to give a location on the screen where you can click and/or
buttons on the keyboard you can press in order to take action. And it turns out that with actually not all that
much additional training, the models can get
quite good at that task. It's a good example of generalization. You know, people sometimes say, if you get to lower earth orbit, you're like halfway to anywhere, right? Because of how much it
takes to escape the gravity. Well, if you have a
strong pre-trained model, I feel like you're halfway to anywhere in terms of the intelligence space. And so actually, it
didn't take all that much to get Claude to do this. And you can just set that in a loop, give the model a screenshot,
tell it what to click on, give it the next screenshot,
tell it what to click on, and that turns into a full kind of almost 3D video
interaction of the model. And it's able to do all
of these tasks, right? You know, we showed these demos where it's able to like
fill out spreadsheets. It's able to kind of like
interact with a website. It's able to, you know, it's able to open all kinds
of, you know, programs, different operating systems,
Windows, Linux, Mac. So, you know, I think all
of that is very exciting. I will say, while in theory, there's nothing you could do there that you couldn't have done through just giving the model the API
to drive the computer screen. This really lowers the barrier. And you know, there's a
lot of folks who either, you know, aren't in a position to interact with those APIs or it takes
them a long time to do. It's just the screen is
just a universal interface that's a lot easier to interact with. And so I expect over time, this is gonna lower a bunch of barriers. Now, honestly, the current model has, it leaves a lot still to be desired, and we were honest about
that in the blog, right? It makes mistakes, it misclicks. And, you know, we were
careful to warn people, hey, this thing isn't, you
can't just leave this thing to, you know, run on your computer
for minutes and minutes. You gotta give this thing
boundaries and guardrails. And I think that's one of the reasons we released it first in an API form rather than kind of, you know, this kind of just hand it to the consumer and give it control of their computer. But you know, I definitely
feel that it's important to get these capabilities out there. As models get more powerful, we're gonna have to
grapple with, you know, how do we use these capabilities safely? How do we prevent them from being abused? And you know, I think releasing the model while the capabilities are, you know, are still limited is very
helpful in terms of doing that. You know, I think since
it's been released, a number of customers, I think Replit was maybe
one of the most quickest to deploy things. You know, have made use
of it in various ways. People have hooked up demos for, you know, Windows desktops, Macs,
you know, Linux machines. So yeah, it's been very exciting. I think as with anything else, you know, it comes with new exciting abilities and then, you know, with those new exciting abilities, we have to think about how to, you know, make the model, you know, safe, reliable, do what humans want them to do. I mean, it's the same story
for everything, right? Same thing. It's that same tension. - But the possibility of use cases here, just the range is incredible. So how much to make it work
really well in the future? How much do you have to specially kind of go beyond what's the
pre-trained model's doing, do more post-training, RLHF
or supervised fine tuning, or synthetic data just
for the agentic stuff? - Yeah, I think speaking at a high level, it's our intention to
keep investing a lot in, you know, making the model better. Like I think, you know,
we look at some of the, you know, some of the benchmarks where previous models were like, oh, could do it 6% of the time, and now our model would do
it 14 or 22% of the time. And yeah, we wanna get up to, you know, the human level reliability of 80, 90% just like anywhere else, right? We're on the same curve that
we were on with SWE-bench, where I think I would
guess a year from now, the models can do this
very, very reliably, but you gotta start somewhere. - So you think it's possible to get to the human level, 90%, basically doing the same
thing you're doing now? Or it has to be special for computer use? - I mean, it depends what
you mean by, you know, special and special in general. But, you know, I
generally think, you know, the same kinds of techniques that we've been using to
train the current model, I expect that doubling
down on those techniques in the same way that we have for code for models in general, you know, for image input, you know, for voice. I expect those same
techniques will scale here, as they have everywhere else. - But this is giving sort of the power of action to Claude, and so you could do a lot
of really powerful things, but you could do a lot of damage also. - Yeah, yeah, no, and we've
been very aware of that. Look, my view actually is computer use isn't a fundamentally new capability, like the CBRN or autonomy
capabilities are. It's more like, it kind of
opens the aperture for the model to use and apply its existing abilities. And so the way we think about it, going back to our RSP, is nothing that this model is
doing inherently increases, you know, the risk from
an RSP perspective. But as the models get more
powerful, having this capability may make it scarier once it, you know, once it has the cognitive
capability to, you know, to do something at the ASL
three and ASL four level, you know, this may be the thing that kind of unbounds it from doing so. So, going forward, certainly
this modality of interaction is something we have tested for, and that we will continue to
test for in RSP going forward. I think it's probably better to have, to learn and explore this capability before the model is super,
you know, super capable. - Yeah, and there's a lot
of interesting attacks, like prompt injection, because now you've widened the aperture, so you can prompt inject
through stuff on screen. So if this becomes more and more useful, then there's more and more benefit to inject stuff into the model. If it goes to a certain webpage, it could be harmless
stuff like advertisements or it could be like harmful stuff, right? - Yeah, I mean, we've
thought a lot about things like spam, CAPTCHA, you know, mass camp. There's all, you know, every, like one secret I'll tell you, if you've invented a new technology, not necessarily the biggest misuse, but the first misuse you'll see, scams, just petty scams. Like you'll just, it's
like a thing as old, people scamming each other, it's this thing as old as time, and it's just every time
you gotta deal with it. - It's almost like silly
to say but it's true, sort of bots and spam
in general is a thing as it gets more and more intelligent. It's harder and harder to fight. - There are a lot of like I said, like there are a lot of
petty criminals in the world. And you know, it's like
every new technology is like a new way for petty
criminals to do something, you know, something stupid and malicious. - Is there any ideas about sandboxing it? Like how difficult is the sandboxing task? - Yeah, we sandbox during training. So for example, during training, we didn't expose the
model to the internet. I think that's probably a
bad idea during training because, you know, the model
can be changing its policy, it can be changing what it's doing, and it's having an
effect in the real world. You know, in terms of actually
deploying the model, right, it kind of depends on the application. Like, you know, sometimes
you want the model to do something in the real world, but of course, you can always put guardrails on the outside, right? You can say, okay, well, you know, this model's not gonna move
data from my, you know, model's not gonna move
any files from my computer or my web server to anywhere else. Now, when you talk about sandboxing, again, when we get to ASL four, none of these precautions are going to make sense there, right, where when you talk about ASL four, you're then, the model is
being kind of, you know, there's a theoretical worry the model could be smart enough to kind of break out of any box. And so there we need to think about mechanistic interpretability
about, you know, if we're gonna have a sandbox, it would need to be a
mathematically provable sand. You know, that's a whole different world than what we're dealing
with with the models today. - Yeah, the science of building a box from which ASL four AI
system cannot escape. - I think it's probably
not the right approach. I think the right approach
instead of having something, you know, unaligned
that like you're trying to prevent it from escaping. I think it's better to
just design the model the right way or have a loop where, you know, you look inside,
you look inside the model and you're able to verify properties, and that gives you an opportunity to like iterate and actually get it right. I think containing bad models is much worse solution
than having good models. - Let me ask about regulation. What's the role of regulation
in keeping AI safe? So for example, can you describe
California AI regulation Bill SB 1047 that was ultimately
vetoed by the governor? What are the pros and cons
of this bill in general? - Yes, we ended up making some suggestions to the bill, and then
some of those were adopted and, you know, we felt,
I think quite positively, quite positively about the
bill by the end of that. It did still have some downsides, and, you know, of course it got vetoed. I think at a high level, I
think some of the key ideas behind the bill are, you know, I would say similar to
ideas behind our RSPs. And I think it's very important
that some jurisdiction, whether it's California
or the federal government and/or other countries and other states passes
some regulation like this. And I can talk through why
I think that's so important. So I feel good about our RSP. It's not perfect, it needs
to be iterated on a lot, but it's been a good forcing function for getting the company to
take these risks seriously, to put them into product
planning, to really make them a central part of work at Anthropic and to make sure that all of 1000 people, and it's almost 1000
people now at Anthropic, understand that this is one
of the highest priorities of the company, if not
the highest priority. But one, there are still some companies that don't have RSP like mechanisms, like OpenAI, Google did
adopt these mechanisms a couple months after Anthropic did, but there are other companies out there that don't have these mechanisms at all. And so if some companies adopt these mechanisms and others don't, it's really gonna create
a situation where, you know, some of these
dangers have the property that it doesn't matter
if three out of five of the companies are being safe, if the other two are being unsafe, it creates this negative externality. And I think the lack of
uniformity is not fair to those of us who have
put a lot of effort into being very thoughtful
about these procedures. The second thing is, I don't think you can
trust these companies to adhere to these voluntary
plans in their own, right? I like to think that Anthropic will. We do everything we can that we will. Our RSP is checked by our
long-term benefit trust. So, you know, we do everything we can to adhere to our own RSP. But you know, you hear lots of things about various companies saying, oh, they said they would give this much compute and they didn't. They said they would do
this thing and they didn't. You know, I don't think it makes sense to, you know, to litigate particular things that companies have done. But I think this broad principle that like if there's
nothing watching over them, there's nothing watching
over us as an industry, there's no guarantee that
we'll do the right thing, and the stakes are very high. And so I think it's important to have a uniform standard
that everyone follows, and to make sure that simply
that the industry does what a majority of the industry has already said is important
and has already said that they definitely will do. Right, some people, you know, I think there's a class of people who are against regulation on principle. I understand where that comes from. If you go to Europe and, you know, you see something like GDPR, you see some of the other
stuff that they've done. You know, some of it's good, but some of it is really
unnecessarily burdensome, and I think it's fair to say really has slowed innovation. And so I understand where people
are coming from on priors. I understand why people come from, start from that position. But again, I think AI is different. If we go to the very
serious risks of autonomy and misuse that I talked about, you know, just a few minutes ago, I think that those are unusual and they warrant an
unusually strong response. And so I think it's very important. Again, we need something
that everyone can get behind. You know, I think one of
the issues with SB 1047, especially the original version of it, was it had a bunch of
the structure of RSPs, but it also had a bunch of
stuff that was either clunky or that just would've created a bunch of burdens, a bunch of hassle, and might even have missed the target in terms of addressing the risks. You don't really hear about it on Twitter. You just hear about kind of, you know, people are cheering for any regulation, and then the folks who are against make up these often quite
intellectually dishonest arguments about how, you know, it'll make us move away from California. Bill doesn't apply if you're
headquartered in California, bill only applies if you
do business in California. Or that it would damage
the open source ecosystem, or that it would, you know, it would cause all of these things. I think those were mostly nonsense, but there are better
arguments against regulation. There's one guy, Dean Ball,
who's really, you know, I think a very scholarly,
scholarly analyst who looks at what happens when
a regulation is put in place and ways that they can kind
of get a life of their own, or how they can be poorly designed. And so our interest has always been, we do think there should be
regulation in this space, but we wanna be an actor who makes sure that that regulation is
something that's surgical, that's targeted at the serious risks and is something people
can actually comply with. Because something I think the advocates of regulation don't understand
as well as they could is if we get something in place that's poorly targeted, that wastes a bunch of people's time, what's gonna happen is
people are gonna say, see, these safety risks, you know, this is nonsense. You know, I just had to hire 10 lawyers, you know, to fill out all these forms. I had to run all these tests for something that was clearly not dangerous. And after six months of that, there will be a groundswell
and we'll end up with a durable consensus
against regulation. And so I think the worst enemy of those who want real accountability
is badly designed regulation. We need to actually get it right. And this is, if there's
one thing I could say to the advocates, it
would be that I want them to understand this dynamic better, and we need to be really careful and we need to talk to people who actually have experience seeing how regulations
play out in practice. And the people who have
seen that understand to be very careful. If this was some lesser issue, I might be against regulation at all. But what I want the
opponents to understand is that the underlying
issues are actually serious. They're not something that
I or the other companies are just making up because
of regulatory capture. They're not sci-fi fantasies. They're not any of these things. You know, every time we have a new model, every few months, we measure
the behavior of these models and they're getting better and better at these concerning tasks, just as they are getting
better and better at, you know, good, valuable,
economically useful tasks. And so I would just love
it if some of the former, you know, I think SB
1047 was very polarizing. I would love it if some of
the most reasonable opponents and some of the most reasonable proponents would sit down together. And, you know I think that, you know, the different AI companies, you know, Anthropic was the only AI
company that, you know, felt positively in a very detailed way. I think Elon tweeted
briefly something positive. But, you know, some of the big ones, like Google, OpenAI, Meta, Microsoft were pretty staunchly against. So I would really like is if, you know, some of the key stakeholders, some of the, you know,
most thoughtful proponents and some of the most thoughtful
opponents would sit down and say, how do we solve
this problem in a way that the proponents feel brings
a real reduction in risk, and that the opponents feel that it is not hampering the industry or hampering innovation any
more necessary than it needs to. And I think for whatever reason that things got too polarized and those two groups
didn't get to sit down in the way that they should. And I feel urgency. I really think we need
to do something in 2025. You know, if we get to the end of 2025 and we've still done nothing about this, then I'm gonna be worried. I'm not worried yet, because again, the risks aren't here yet, but I think time is running short. - Yeah, and come up with
something surgical, like you said. - Yeah, yeah, yeah, exactly. And we need to get away from this intense pro-safety versus intense anti-regulatory
rhetoric, right? It's turned into these
flame wars on Twitter and nothing good's gonna come of that. - So there's a lot of curiosity about the different players in the game. One of the OGs is OpenAI. You've had several years
of experience at OpenAI. What's your story and history there? - Yeah, so I was at OpenAI
for roughly five years. For the last, I think it was couple years, you know, I was vice
president of research there. Probably myself and Ilya
Sutskever were the ones who, you know, really kind of
set the research direction. Around 2016 or 2017, I first
started to really believe in or at least confirm my belief
in the Scaling Hypothesis when Ilya famously said to me, "The thing you need to understand about these models is
they just wanna learn. The models just wanna learn." And again, sometimes there
are these one sentences, these zen koans that you hear them and you're like, ah,
that explains everything. That explains like 1000
things that I've seen. And then I, you know, ever after I had this
visualization in my head of like, you optimize the models in the right way, you point the models in the right way. They just wanna learn. They just wanna solve the problem, regardless of what the problem is. - So get out of their way, basically. - Get out of their way, yeah. Don't impose your own ideas
about how they should learn. And you know, this was the same thing as Rich Sutton put out
in the Bitter Lesson or Gwern put out in
The Scaling Hypothesis. You know, I think generally
the dynamic was, you know, I got this kind of inspiration
from Ilya and from others, folks like Alec Radford
who did the original GPT-1, and then ran really hard with it. Me and my collaborators on GPT-2, GPT-3. RL from Human Feedback,
which was an attempt to kind of deal with the
early safety and durability. Things like debate and amplification. Heavy on interpretability. So again, the combination
of safety plus scaling. Probably 2018, 2019, 2020, those were kind of the years when myself and my collaborators
probably, you know, many of whom became
co-founders of Anthropic, kind of really had a vision
and like drove the direction. - Why'd you leave? Why'd you decide to leave? - Yeah, so look, I'm
gonna put things this way and, you know, I think it ties to the race to the top, right? Which is, you know, in my time at OpenAI, what I'd come to see is
I'd come to appreciate the Scaling Hypothesis, and as I'd come to appreciate
kind of the importance of safety along with
the Scaling Hypothesis. The first one I think, you know, OpenAI was getting on board with. The second one in a way
had always been part of OpenAI's messaging, but, you know, over many years of the time that I spent there, I think I had a particular vision of how we should handle these things, how we should be brought out in the world, the kind of principles that
the organization should have. And look, I mean, there were
like many, many discussions about like, you know, should the org do, should the company do this? Should the company do that? Like, there's a bunch of
misinformation out there. People say like, we left because we didn't like
the deal with Microsoft. False, although, you know, it
was like a lot of discussion, a lot of questions about exactly how we do the deal with Microsoft. We left because we didn't
like commercialization. That's not true, we built GPT-3, which was the model
that was commercialized. I was involved in commercialization. It's more again about, how do you do it? Like civilization is going down this path to very powerful AI. What's the way to do it that is cautious, straightforward, honest, that builds trust in the
organization and individuals? How do we get from here to there? And how do we have a real
vision for how to get it right? How can safety not just
be something we say because it helps with recruiting? And, you know, I think
at the end of the day, if you have a vision for that, forget about anyone else's vision. I don't wanna talk about
anyone else's vision. If you have a vision for how
to do it, you should go off and you should do that vision. It is incredibly unproductive to try and argue with someone else's vision. You might think they're
not doing it the right way. You might think they're dishonest. Who knows, maybe you're
right, maybe you're not. But what you should do is you should take some people you trust and you should go off together and you should make your vision happen. And if your vision is compelling, if you can make it appeal to people, you know, some combination of ethically, you know, in the market, you know, if you can make a company that's
a place people wanna join, that, you know, engages in practices that people think are reasonable, while managing to maintain its position in the ecosystem at the same time, if you do that, people will copy it. And the fact that you are
doing it, especially the fact that you're doing it better than they are causes them to change their behavior in a much more compelling way than if they're your boss
and you're arguing with them. I just, I don't know how to be any more specific about it than that, but I think it's generally
very unproductive to try and get someone else's vision to look like your vision. It's much more productive to go off and do a clean experiment
and say, this is our vision, this is how we're gonna do things. Your choice is you can ignore us, you can reject what we're doing, or you can start to become more like us, and imitation is the
sincerest form of flattery. And you know, that plays out
in the behavior of customers, that plays out in the
behavior of the public, that plays out in the behavior of where people choose to work. And again, at the end, it's not about one company winning or another company winning. If we or another company are
engaging in some practice that, you know, people find genuinely appealing, and I want it to be in substance, not just in appearance. And, you know, I think researchers are sophisticated and
they look at substance. And then other companies
start copying that practice and they win because they
copied that practice, that's great, that's success. That's like the race to the top. It doesn't matter who wins in the end, as long as everyone is copying everyone else's good practices, right? One way I think of it is like, the thing we're all afraid of is the race to the bottom, right? And the race to the bottom, doesn't matter who wins
because we all lose, right? Like, you know, in the most extreme world, we make this autonomous AI that, you know, the robots enslave us or whatever, right? I mean, that's half joking, but you know, that is the most extreme
thing that could happen. Then it doesn't matter
which company was ahead. If instead you create a race to the top where people are competing to engage in good practices, then, you know, at the end of the day, you know, it doesn't matter who ends up winning, doesn't even matter who
started the race at the top. The point isn't to be virtuous. The point is to get the system into a better equilibrium
than it was before, and individual companies can
play some role in doing this. Individual companies can, you know, can help to start it, can help to accelerate it. And frankly, I think
individuals at other companies have done this as well, right? The individuals that
when we put out an RSP react by pushing harder to
get something similar done, get something similar
done at other companies. Sometimes other companies
do something that's like, we're like, oh, it's a good practice. We think that's good. We should adopt it too. The only difference is,
you know, I think we are, we try to be more forward-leaning. We try and adopt more
of these practices first and adopt them more quickly
when others invent them. But I think this dynamic is
what we should be pointing at. And I think it abstracts
away the question of, you know, which company's
winning, who trusts who. I think all these questions of drama are profoundly uninteresting, and the thing that
matters is the ecosystem that we all operate in and how
to make that ecosystem better because that constrains all the players. - And so Anthropic is this kind of clean experiment built on a foundation of like what concretely AI
safety should look like. - Look, I'm sure we've made plenty of mistakes along the way. The perfect organization doesn't exist. It has to deal with the imperfection of 1000 employees. It has to deal with the imperfection of our leaders, including me. It has to deal with the imperfection of the people we've put to, you know, to oversee the imperfection
of the leaders, like the board and the
long-term benefit trust. It's all a set of imperfect people trying to aim imperfectly at some ideal that will never perfectly be achieved. That's what you sign up for, that's what it will always be. But imperfect doesn't
mean you just give up. There's better and there's worse. And hopefully we can begin to build, we can do well enough
that we can begin to build some practices that the
whole industry engages in. And then, you know, my guess is that multiple of these companies
will be successful. Anthropic will be successful. These other companies, like ones I've been at the
past will also be successful, and some will be more
successful than others. That's less important than, again, that we align the
incentives of the industry. And that happens partly
through the race to the top, partly through things like RSP, partly through again
selected surgical regulation. - You said talent density
beats talent mass. So can you explain that? Can you expand on that? Can you just talk about what it takes to build a great team of AI
researchers and engineers? - This is one of these statements that's like more true every month. Every month I see this statement as more true than I did the month before. So if I were to do a thought experiment, let's say you have a team of 100 people that are super smart, motivated, and aligned with the mission,
and that's your company. Or you can have a team of 1000 people where 200 people are super smart, super aligned with the mission, and then like 800 people are, let's just say you pick 800 like random big tech employees, which would you rather have, right? The talent mass is greater in the group of 1000 people, right? You have even a larger number of incredibly talented,
incredibly aligned, incredibly smart people. But the issue is just that if every time someone super talented looks around, they see someone else super
talented and super dedicated, that sets the tone for everything, right? That sets the tone for
everyone is super inspired to work at the same place. Everyone trusts everyone else. If you have 1000 or 10,000 people and things have really regressed, right? You are not able to do selection and you're choosing random people, what happens is then you need
to put a lot of processes and a lot of guardrails in place just because people don't
fully trust each other, or you have to adjudicate
political battles. Like there are so many things that slow down the org's
ability to operate. And so we're nearly 1000 people and you know, we've
tried to make it so that as large a fraction of those 1000 people as possible are like super
talented, super skilled. It's one of the reasons we've slowed down hiring a
lot in the last few months. We grew from 300 to 800, I believe, I think in the first seven,
eight months of the year. And now we've slowed down. We're at like, you know,
the last three months, we went from 800 to 900,
950, something like that. Don't quote me on the exact numbers, but I think there's an
inflection point around 1000, and we want to be much
more careful how we grow. Early on, and now as well, you know, we've hired a lot of physicists. You know, theoretical physicists can learn things really fast. Even more recently as we've
continued to hire that, you know, we've really had a high bar for, on both the research side and the software engineering side have hired a lot of senior people, including folks who used to be at other companies in this space. And we've just continued
to be very selective. It's very easy to go from 100 to 1000 and 1000 to 10,000
without paying attention to making sure everyone
has a unified purpose. It's so powerful. If your company consists of
a lot of different fiefdoms that all wanna do their own thing, they're all optimizing
for their own thing, it's very hard to get anything done. But if everyone sees the
broader purpose of the company, if there's trust and there's dedication to doing the right thing,
that is a superpower. That in itself, I think, can overcome almost
every other disadvantage. - And you know, as to
Steve Jobs, A players. A players wanna look around and see other A players is
another way of saying that. I don't know what that
is about human nature, but it is demotivating to see people who are not obsessively driving
towards a singular mission. And it is, on the flip side of that, super motivating to see that. It's interesting. What's it take to be a great AI researcher or engineer from everything you've seen, from working with so many amazing people? - Yeah, I think the number one quality, especially on the research side, but really both is open-mindedness. Sounds easy to be open-minded, right? You're just like, oh,
I'm open to anything. But, you know, if I think about my own early history in
the Scaling Hypothesis, I was seeing the same
data others were seeing. I don't think I was
like a better programmer or better at coming up with research ideas than any of the hundreds of
people that I worked with. In some ways, I was worse. You know, like I've never like, you know, precise programming of like,
you know, finding the bug, writing the GPU kernels. Like, I could point you to 100 people here who are better at that than I am. But the thing that I think I did have that was different was
that I was just willing to look at something with new eyes, right? People said, oh, you know, "We don't have the right algorithms yet. We haven't come up with the
right way to do things." And I was just like, oh, I don't know, like, you know, this neural net has like 30 billion,
30 million parameters. Like, what if we gave
it 50 million instead? Like, let's plot some graphs. Like that basic scientific
mindset of like, oh, man, like I just, like, you know, I see some variable that I could change. Like, what happens when it changes? Like, let's try these different things and like create a graph. For even, this was like the simplest thing in the world, right? Change the number of, you know, this wasn't like PhD
level experimental design. This was like simple and stupid. Like, anyone could have done this if you just told them
that it was important. It's also not hard to understand. You didn't need to be
brilliant to come up with this. But you put the two things together and, you know, some tiny number of people, some single digit number of people have driven forward the whole
field by realizing this. And you know, it's often like that. If you look back at the
discovery, you know, the discoveries in history, they're often like that. And so this open-mindedness and this willingness to see with new eyes that often comes from
being newer to the field. Often experience is a
disadvantage for this. That is the most important thing. It's very hard to look for and test for. But I think it's the most important thing because when you find something, some really new way of
thinking about things, when you have the initiative to do that, it's absolutely transformative. - And also be able to do kind
of rapid experimentation, and in the face of that,
be open-minded and curious and looking at the data,
just these fresh eyes and seeing what is that
it's actually saying. That applies in mechanism
interpretability. - It's another example of this. Like some of the early work in mechanistic
interpretability, so simple, it's just no one thought to
care about this question before. - You said what it takes to
be a great AI researcher. Can we rewind the clock back? What advice would you give
to people interested in AI? They're young, looking forward to, how can I make an impact on the world? - I think my number one piece of advice is to just start playing with the models. This was actually, I worry a little, this seems like obvious advice now. I think three years ago, it wasn't obvious and people started by, oh, let me read the latest
Reinforcement Learning paper. Let me, you know, let me kind of, I mean, that was really, and I mean, you should do that as well. But now, you know, with wider availability of models and APIs, people
are doing this more. But I think just experiential knowledge. These models are new artifacts that no one really understands, and so getting experience
playing with them. I would also say, again,
in line with the like, do something new, think
in some new direction. Like there are all these things
that haven't been explored. Like for example, mechanistic interpretability
is still very new. It's probably better to work on that than it is to work on
new model architectures because, you know, it's more
popular than it was before. There are probably like
100 people working on it, but there aren't like
10,000 people working on it. And it's just this fertile area for study. Like, you know, there's so
much like low hanging fruit. You can just walk by and, you know, you can just walk by
and you can pick things. And the only reason, for whatever reason, people aren't interested in it enough. I think there are some things around long horizon learning
and long horizon tasks where there's a lot to be done. I think evaluations are still, we're still very early in our
ability to study evaluations, particularly for dynamic
systems acting in the world. I think there's some
stuff around multi-agent. Skate where the puck
is going is my advice. And you don't have to be
brilliant to think of it. Like all the things that
are gonna be exciting in five years, like people
even mention them as like, you know, conventional wisdom, but like, it's just somehow
there's this barrier that people don't double down
as much as they could, or they're afraid to do something that's not the popular thing. I don't know why it happens, but like, getting over that barrier, that's the my number one piece of advice. - Let's talk if we could
a bit about post-training. So it seems that the
modern post-training recipe has a little bit of everything. So supervised fine tuning, RLHF, the Constitutional AI with RLAIF. - Best acronym. - It's, again, that name thing. - [Dario] RLAIF. (both laughing) - And then synthetic data, seems like a lot of synthetic data, or at least trying to figure out ways to have high quality synthetic data. So what's the, if this is a secret sauce that makes Anthropic Claude so incredible, how much of the magic
is in the pre-training? How much of is in the post-training? - Yeah, I mean, so first of all, we're not perfectly able
to measure that ourselves. - [Lex] True. - You know, when you see
some great character ability, sometimes it's hard to tell whether it came from
pre-training or post-training. We've developed ways to try and distinguish between those two but they're not perfect. You know, the second thing
I would say is, you know, when there is an advantage, and I think we've been pretty good in general at RL, perhaps the best. Although I don't know 'cause I don't see what goes on inside other companies. Usually it isn't, oh my God, we have this secret magic method that others don't have, right? Usually it's like, well, you know, we got better at the infrastructure, so we could run it for longer. Or, you know, we were able
to get higher quality data, or we were able to filter our data better, or we were able to, you know, combine these methods in practice. It's usually some boring matter of kind of practiced and trade craft. So, you know, when I think about how to do something special in terms of how we train these
models, both pre-training, but even more so post-training, you know, I really think of it a little more, again, as like designing airplanes or cars. Like, you know, it's
not just like, oh, man, I have the blueprint. Like maybe that makes you
make the next airplane. But like, there's some
cultural trade craft of how we think about the design process that I think is more
important than, you know, than any particular gizmo
we're able to invent. - Okay, well, let me ask you
about specific techniques. So first on RLHF, why do
you think, just zooming out, intuition almost philosophy, why do you think RLHF works so well? - If I go back to like
the Scaling Hypothesis, one of the ways to skate
the Scaling Hypothesis is if you train for X and you throw enough compute
at it, then you get X. And so RLHF is good at doing what humans want the model to do, or at least to state it more precisely, doing what humans who look at the model for a brief period of time and consider different possible responses, what they prefer as the response, which is not perfect from both the safety and capabilities perspective, in that humans are often not
able to perfectly identify what the model wants, and what humans want in
the moment may not be what they want in the long term. So there's a lot of subtlety there, but the models are good at, you know, producing what the humans
in some shallow sense want. And it actually turns out
that you don't even have to throw that much compute at
it because of another thing, which is this thing about
a strong pre-trained model being halfway to anywhere. So once you have the pre-trained model, you have all the representations you need to get the model where you want it to go. - So do you think RLHF
makes the model smarter or just appears smarter to the humans? - I don't think it
makes the model smarter. I don't think it just makes
the model appear smarter. It's like RLHF like bridges the gap between the human and the model, right? I could have something really smart that like can't communicate at all, right? We all know people like this,
people who are really smart, but that, you know, you can't understand what they're saying. So I think RLHF just bridges that gap. I think it's not the
only kind of RL we do, it's not the only kind of RL
that will happen in the future. I think RL has the potential
to make models smarter, to make them reason better,
to make them operate better, to make them develop new skills even. And perhaps that could be done, you know, even in some cases with human feedback. But the kind of RLHF we do today mostly doesn't do that yet, although we're very quickly
starting to be able to. - But it appears to sort of increase, if you look at the metric of helpfulness, it increases that. - It also increases, what was this word in Leopold's essay, unhobbling, where basically the models are hobbled and then you do various trainings
to them to unhobble them. So, you know, I like that word 'cause it's like a rare word. But so I think RLHF unhobbles
the models in some ways. And then there are other ways where that model hasn't yet been unhobbled and, you know, needs to unhobble. - If you can say in terms of cost, is pre-training the most expensive thing? Or is post-training creep up to that? - At the present moment, it is still the case that pre-training is the majority of the cost. I don't know what to expect in the future, but I could certainly anticipate a future where post-training is
the majority of the cost. - In that future you anticipate, would it be the humans or the AI that's the cost of thing
for the post-training? - I don't think you can
scale up humans enough to get high quality. Any kind of method that relies on humans and uses a large amount of compute, it's gonna have to rely on some
scaled superposition method, like you know, debate or
iterated amplification or something like that. - So on that super
interesting set of ideas around Constitutional AI, can you describe what it is, as first detailed in December 2022 paper and beyond that, what is it? - Yes, so this was from two years ago. The basic idea is, so we
describe what RLHF is. You have a model and it,
you know, spits out two, you know, like you just
sample from it twice, it spits out two possible responses, and you're like, "Human, which
response do you like better?" Or another variant of it is, "Rate this response on a
scale of one to seven." So that's hard because you need to scale up human interaction
and it's very implicit, right? I don't have a sense of
what I want the model to do. I just have a sense of
like what this average of a 1000 humans wants the model to do. So two ideas. One is, could the AI system itself decide which response is better, right? Could you show the AI
system these two responses and ask which response is better? And then second, well, what
criterion should the AI use? And so then there's this idea, 'cause you have a single document, a constitution, if you will, that says, these are the principles the model should be using to respond. And the AI system reads those, it reads those principles, as well as reading the
environment and the response. And it says, well, how
good did the AI model do? It's basically a form of self play. You're kind of training
the model against itself. And so the AI gives the response and then you feed that back into what's called the preference model, which in turn feeds the
model to make it better. So you have this triangle of like the AI, the preference model, and the
improvement of the AI itself. - And we should say that
in the constitution, the set of principles are
like human interpretable. They're like-
- Yeah, yeah. It's something both the human
and the AI system can read. So it has this nice kind of
translatability or symmetry. You know, in practice we
both use a model constitution and we use RLHF and we use
some of these other methods. So it's turned into one tool in a toolkit that both reduces the need for RLHF and increases the value we get from using each data point of RLHF. It also interacts in interesting ways with kind of future
reasoning type RL methods. So it's one tool in the toolkit, but I think it is a very important tool. - Well, it's a compelling
one to us humans. You know, thinking about
the founding fathers and the founding of the United States, the natural question is, who and how do you think it
gets to define the constitution, the set of principles in the constitution? - Yeah, so I'll give
like a practical answer and a more abstract answer. I think the practical
answer is like, look, in practice models get used by all kinds of different like customers, right? And so you can have this
idea where, you know, the model can have specialized
rules or principles. You know, we fine tune
versions of models implicitly. We've talked about doing it explicitly, having special principles that people can build into the models. So from a practical perspective, the answer can be very
different from different people. You know, customer
service agent, you know, behaves very differently from a lawyer and obeys different principles. But I think at the base of it,
there are specific principles that models, you know, have to obey. I think a lot of them are things that people would agree with. Everyone agrees that, you know, we don't want models to
present these CBRN risks. I think we can go a little further and agree with some basic principles of democracy in the rule of law. Beyond that, it gets,
you know, very uncertain, and there, our goal is
generally for the models to be more neutral, to not espouse a particular point of view, and, you know, more just
be kind of like wise agents or advisors that will help
you think things through and will, you know, present
possible considerations, but, you know, don't express, you know, strong or specific opinions. - OpenAI released a model spec where it kind of clearly,
concretely defines some of the goals of the model, and specific examples, like A/B, how the model should behave. Do you find that interesting? By the way, I should mention, I believe the brilliant John
Schulman was a part of that. He's now at Anthropic. Do you think this is a useful direction? Might Anthropic release
a model spec as well? - Yeah, so I think that's
a pretty useful direction. Again, it has a lot in common
with Constitutional AI. So again, another example of
like a race to the top, right? We have something that's like we think, you know, a better and more
responsible way of doing things. It's also a competitive advantage. Then others kind of, you know, discover that it has advantages and then start to do that thing. We then no longer have
the competitive advantage, but it's good from the perspective that now everyone has
adopted a positive practice that others were not adopting. And so our response to that is, well, looks like we need a new
competitive advantage in order to keep driving
this race upwards. So that's how I generally feel about that. I also think every implementation of these things is different. So, you know, there were some things in the model spec that were
not in Constitutional AI, and so, you know, we can
always adopt those things or, you know, at least learn from them. So again, I think this is an example of like the positive dynamic that I think we should all
want the field to have. - Let's talk about the incredible essay "Machines of Loving Grace." I recommend everybody read it. It's a long one. - It is rather long.
- Yeah. It's really refreshing
to read concrete ideas about what a positive future looks like. And you took sort of a bold stance because like, it's very possible that you might be wrong on the dates or specific applications.
- Oh, yeah. I'm fully expecting to, you know, will definitely be wrong
about all the details. I might be just spectacularly wrong about the whole thing and people will, you know, will laugh at me for years. That's just how the future works. (laughs) - So you provided a bunch of concrete positive impacts of AI and how, you know, exactly a super intelligent
AI might accelerate the rate of breakthroughs in, for example, biology and chemistry that would then lead to things like we cure most cancers, prevent all infectious disease, double the human lifespan and so on. So let's talk about this essay. First, can you give a high
level vision of this essay and what key takeaways
that people would have? - Yeah, I have spent a lot of time, and Anthropic has spent a lot
of effort on like, you know, how do we address the risks of AI, right? How do we think about those risks? Like we're trying to do a
race to the top, you know, that requires us to build
all these capabilities and the capabilities are cool, but you know, we're like, a big part of what we're trying to do is like address the risks. And the justification for that is like, well, you know, all these positive things, you know, the market is this
very healthy organism, right? It's gonna produce all
the positive things. The risks, I don't know, we might mitigate them, we might not. And so we can have more impact by trying to mitigate the risks. But I noticed that one flaw
in that way of thinking, and it's not a change in how
seriously I take the risks. It's maybe a change in
how I talk about them. Is that, you know, no
matter how kind of logical or rational that line of reasoning that I just gave might be, if you kind of only talk about risks, your brain only thinks about risks. And so I think it's
actually very important to understand, what if things do go well? And the whole reason we're
trying to prevent these risks is not because we're afraid of technology, not because we wanna slow it down. It's because if we can get to the other side of these risks, right? If we can run the gauntlet successfully, you know, to put it in stark terms, then on the other side of the gauntlet are all these great things and these things are worth fighting for, and these things can
really inspire people. And I think I imagine, because look, you have all
these investors, all these VCs, all these AI companies talking about all the positive benefits of AI. But as you point out, it's weird, there's actually a dearth of really getting specific about it. There's a lot of like
random people on Twitter like posting these kind
of like gleaning cities, and this just kind of
like vibe of like grind, accelerate harder, like
kick out the, you know, it's just this very like
aggressive ideological. But then you're like, well, what are you actually excited about? And so I figured that, you know, I think it would be
interesting and valuable for someone who's actually coming from the risk side to try, and to try and really
make a try at explaining what the benefits are, both because I think it's
something we can all get behind and I want people to understand. I want them to really understand that this isn't doomers versus accelerationist. This is that, if you
have a true understanding of where things are going with with AI, and maybe that's the more important axis. AI is moving fast versus
AI is not moving fast, then you really appreciate the benefits and you really, you want humanity, our civilization to seize those benefits, but you also get very serious about anything that could derail them. - So I think the starting point is to talk about what this powerful AI, which is the term you like to use, most of the world uses AGI,
but you don't like the term because it's basically
has too much baggage, it's become meaningless. It's like we're stuck with the terms, whether we like them or not.
- Maybe we're stuck with the terms and my efforts
to change them are futile. - It's admirable.
- I'll tell you what else I don't, this is like
a pointless semantic point, but I keep talking about it in public- - Back to naming again.
- So I'm just gonna do it once more. I think it's a little like,
let's say it was like 1995 and Moore's law is making
the computers faster. And like for some reason, there had been this like
verbal tick that like, everyone was like, well,
someday we're gonna have like super computers
and like supercomputers are gonna be able to do
all these things that like, you know, once we have supercomputers, we'll be able to like sequence the genome, we'll be able to do other things. And so, like one, it's true, the computers are getting
faster, and as they get faster, they're gonna be able to
do all these great things. But there's no discreet point at which you had a supercomputer and previous computers were not. Like supercomputer is a term we use, but like, it's a vague
term to just describe like computers that are faster
than what we have today. There's no point at which
you pass the threshold and you're like, oh my God,
we're doing a totally new type of computation and new. And so I feel that way about AGI like, there's just a smooth exponential and like if by AGI you mean like AI is getting better and better, and like gradually, it's gonna do more and more of what humans do until it's gonna be smarter than humans, and then it's gonna get
smarter even from there then yes, I believe in AGI. But if AGI is some
discreet or separate thing, which is the way people
often talk about it, then it's kind of a meaningless buzzword. - Yeah, I mean, to me it's
just sort of a platonic form of a powerful AI, exactly
how you define it. I mean, you define it very nicely. So on the intelligence axis, just on pure intelligence, it's smarter than a Nobel Prize winner, as you describe, across
most relevant disciplines. So, okay, that's just intelligence. So it's both in creativity and be able to generate new ideas, all that kind of stuff, in every discipline, Nobel Prize winner, okay, in their prime. (laughs) It can use every modality, so that's kind of self-explanatory, but just to operate across all
the modalities of the world. It can go off for many hours,
days and weeks to do tasks, and do its own sort of detailed planning and only ask you help when it's needed. It can use, this is
actually kind of interesting because I think in the essay you said, I mean, again, it's a bet, that
it's not gonna be embodied, but it can control embodied tools. So it can control tools,
robots, laboratory equipment. The resources used to train
it can then be repurposed to run millions of copies of it. And each of those copies
would be independent, they could do their own independent work. So you can do the cloning of the intelligence system software. - Yeah, yeah, I mean, you might imagine from outside the field that like, there's only one of these, right? That like, you made it,
you've only made one. But the truth is that like,
the scale up is very quick. Like we do this today, we make a model, and then we deploy thousands, maybe tens of thousands
of instances of it. I think by the time, you know, certainly within two to three years, whether we have these
super powerful AIs or not, clusters are gonna get to the size where you'll be able to
deploy millions of these and they'll be, you
know, faster than humans. And so if your picture is, oh, we'll have one and it'll
take a while to make them. My point, there was no, actually you have millions
of them right away. - And in general they can learn and act 10 to 100 times faster than humans. So that's a really nice
definition of powerful AI, okay. So that, but you also write that clearly such an
entity would be capable of solving very difficult
problems very fast, but it is not trivial
to figure out how fast. Two extreme positions
both seem false to me. So the singularity is on the one extreme and the opposite on the other extreme. Can you describe each of the extremes? - Yeah, so.
- And why. - So yeah, let's describe the extreme. So like one extreme would be, well, look, you know, if we look at kind
of evolutionary history, like there was this big
acceleration where, you know, for hundreds of thousands of years, we just had like, you know,
single celled organisms, and then we had mammals,
and then we had apes, and then that quickly turned to humans. Humans quickly built
industrial civilization. And so this is gonna keep speeding up and there's no ceiling at the human level. Once models get much,
much smarter than humans, they'll get really good at
building the next models, and you know, if you write down like a simple differential equation, like this is an exponential. And so what's gonna happen is that models will build faster models, models will build faster models, and those models will build, you know, nanobots that can like take over the world and produce much more energy than you could produce otherwise. And so if you just kind of like solve this abstract differential equation, then like five days after we, you know, we build the first AI that's more powerful than humans, then, you know, like
the world will be filled with these AIs and every
possible technology that could be invented
like will be invented. I'm caricaturing this a little bit, but, you know, I think that's one extreme. And the reason that I think
that's not the case is that, one, I think they just neglect
like the laws of physics. Like it's only possible to do things so fast in the physical world. Like some of those loops go through, you know, producing faster hardware. It takes a long time to
produce faster hardware. Things take a long time. There's this issue of complexity, like, I think no matter how smart you are, like, you know, people talk about, oh, we can make models
of biological systems that'll do everything
the biological systems. Look, I think computational
modeling can do a lot. I did a lot of computational modeling when I worked in biology, but like, just, there are a lot of things that you can't predict
how they're, you know, they're complex enough
that like just iterating, just running the experiment
is gonna beat any modeling, no matter how smart the
system doing the modeling is. - Well, even if it's not interacting with the physical world, just the modeling is gonna be hard? - Yeah, I think, well, the
modeling's gonna be hard and getting the model to match the physical world is gonna be hard. - All right, so he does have to interact with the physical world to verify. - Yeah, but it's just, you know, you just look at even
the simplest problems. Like, you know, I think I
talk about like, you know, the three body problem or
simple chaotic prediction, like, you know, or like predicting the economy. It's really hard to predict
the economy two years out. Like maybe the case is like,
you know, normal, you know, humans can predict what's gonna happen in the economy next quarter,
or they can't really do that. Maybe a AI system that's, you know, a zillion times smarter
can only predict it out a year or something
instead of, you know. You have these kind of
exponential increase in computer intelligence
for linear increase in ability to predict. Same with, again, like, you know, biological molecules,
molecules interacting. You don't know what's gonna happen when you perturb a complex system. You can find simple parts in it if you're smarter, you're better at finding these simple parts. And then I think human institutions. Human institutions are
just, are really difficult. Like, you know, it's
been hard to get people, I won't give specific examples, but it's been hard to get people to adopt even the technologies
that we've developed, even ones where the
case for their efficacy is very, very strong. You know, people have concerns. They think things are conspiracy theories. Like it's just been,
it's been very difficult. It's also been very
difficult to get, you know, very simple things through
the regulatory system, right? I think, and you know, I don't
wanna disparage anyone who, you know, works in regulatory
systems of any technology. There are hard trade-offs
they have to deal with. They have to save lives. But the system as a whole I think makes some obvious trade-offs that are very far from
maximizing human welfare. And so if we bring AI systems into this, you know, into these human systems, often the level of intelligence may just not be the
limiting factor, right? It just may be that it takes
a long time to do something. Now, if the AI system
circumvented all governments, if it just said "I'm dictator of the world and I'm gonna do whatever," some of these things it could do. Again, the things having
to do with complexity, I still think a lot of
things would take a while. I don't think it helps that the AI systems can produce a lot of
energy or go to the Moon. Like some people in comments responded to the essay saying the AI system can produce a lot of energy
in smarter AI systems. That's missing the point. That kind of cycle doesn't
solve the key problems that I'm talking about here. So I think a bunch of people
missed the point there. But even if it were completely on the line and, you know, could get around all these human obstacles,
it would have trouble. But again, if you want
this to be an AI system that doesn't take over the world, that doesn't destroy humanity, then basically, you know, it's gonna need to follow basic human laws, right? You know, if we want to
have an actually good world, like we're gonna have to have an AI system that interacts with humans, not one that kind of
creates its own legal system or disregards all the laws or all of that. So as inefficient as these
processes are, you know, we're gonna have to deal with them because there needs to be some popular and democratic legitimacy in how these systems are rolled out. We can't have a small group of people who are developing these systems say this is what's best for everyone, right? I think it's wrong, and I think in practice,
it's not gonna work anyway. So you put all those things together and, you know, we're not gonna, you know, change the world and upload everyone in five minutes. I just, I don't think it, A, I don't think it's gonna happen, and B, you know, to the extent that it could happen, it's not the way to lead to a good world. So that's on one side. On the other side, there's
another set of perspectives, which I have actually in
some ways more sympathy for, which is, look, we've seen big productivity increases before, right? You know, economists are familiar with studying the productivity increases that came from the computer revolution and internet revolution. And generally, those
productivity increases were underwhelming. They were less than you might imagine. There was a quote from Robert Solow, "You see the computer
revolution everywhere except the productivity statistics." So why is this the case? People point to the structure of firms, the structure of enterprises. You know, how slow it's been to roll out our existing technology to
very poor parts of the world, which I talk about in the essay, right? How do we get these technologies to the poorest parts of the world that are behind on cell phone technology, computers, medicine, let alone, you know, newfangled AI that
hasn't been invented yet. So you could have a
perspective that's like, well, this is amazing technically,
but it's all a nothing burger. You know, I think Tyler Cowen, who wrote something in response to my essay, has that perspective. I think he thinks the radical
change will happen eventually, but he thinks it'll take 50 or 100 years. And you could have even
more static perspectives on the whole thing. I think there's some truth to it. I think the timescale is just too long. And I can see it, I can actually see both sides with today's AI. So, you know, a lot of our
customers are large enterprises who are used to doing
things a certain way. I've also seen it in talking
to governments, right? Those are prototypical,
you know, institutions, entities that are slow to change. But the dynamic I see over
and over again is, yes, it takes a long time to move the ship. Yes, there's a lot of resistance
and lack of understanding. But the thing that makes
me feel that progress will in the end happen moderately fast, not incredibly fast, but moderately fast, is that you talk to, what I find is I find over and over again, again, in large companies,
even in governments, which have been actually
surprisingly forward-leaning, you find two things that
move things forward. One, you find a small fraction
of people within a company, within a government who
really see the big picture, who see the whole Scaling Hypothesis, who understand where AI is
going, or at least understand where it's going within their industry. And there are a few people like that within the current US government who really see the whole picture. And those people see that this is the most important
thing in the world, and so they agitate for it. And the thing, they alone
are not enough to succeed because they're a small set of people within a large organization. But as the technology starts to roll out, as it succeeds in some places, in the folks who are
most willing to adopt it, the specter of competition
gives them a wind at their backs because they can point within
their large organization, they can say, look, these other
guys are doing this, right? You know, one bank can say, look, this newfangled hedge
fund is doing this thing. They're going to eat our lunch. In the US, we can say we're afraid China's gonna get there before we are. And that combination, the
specter of competition plus a few visionaries within these, you know, within the organizations that in many ways are sclerotic, you put those two things together and it actually makes something happen. I mean, it's interesting. It's a balanced fight between the two because inertia is very powerful. But eventually over enough time, the innovative approach breaks through. And I've seen that happen. I've seen the arc of
that over and over again. And it's like the barriers are there. The barriers to progress, the complexity, not knowing how to use the model or how to deploy them are there, and for a bit, it seems like
they're gonna last forever, like change doesn't happen. But then eventually change happens and always comes from a few people. I felt the same way when I was an advocate of the Scaling Hypothesis
within the AI field itself and others didn't get it. It felt like no one would ever get it. Then it felt like we had a
secret almost no one ever had, and then a couple years later,
everyone has the secret. And so I think that's how it's gonna go with deployment to AI in the world. It's gonna, the barriers are
gonna fall apart gradually and then all at once. And so I think this is gonna be more, and this is just an instinct. I could easily see how I'm wrong. I think it's gonna be more like 5 or 10 years, as I say in the essay, than it's gonna be 50 or 100 years. I also think it's gonna be 5 or 10 years more than it's gonna be, you know, 5 or 10 hours, because I've just seen
how human systems work. And I think a lot of these people who write down these
differential equations who say AI is gonna make more powerful AI, who can't understand how it
could possibly be the case that these things won't change so fast, I think they don't
understand these things. - So what to you is the timeline to where we achieve AGI, AKA powerful AI, AKA super useful AI? - Useful. (laughs) I'm gonna start calling it that. - It's a debate about naming. You know, on pure intelligence, it can smarter than a Nobel Prize winner in every relevant discipline and all the things we've said. Modality, can go and do stuff
on its own for days, weeks, and do biology experiments on its own. In one, you know what,
let's just stick to biology 'cause you sold me on the whole biology and health section,
that's so exciting from, just I was getting giddy from
a scientific perspective. It made me wanna be a biologist. - It's almost, it's so, no, no, this was the feeling I
had when I was writing it that it's like this would
be such a beautiful future if we can just make it happen, right? If we can just get the
landmines out of the way and make it happen, there's so much, there's so much beauty and elegance and moral force
behind it if we can just. And it's something we should
all be able to agree on, right? Like, as much as we fight about all these political questions, is this something that could
actually bring us together? But you were asking when
when will we get this? - When? When do you think? Just so putting numbers on that. - So you know, this is, of course, the thing I've been grappling
with for many years, and I'm not at all confident. Every time, if I say 2026 or 2027, there will be like a zillion like people on Twitter who will be like, "AI CEO said 2026," and it'll be repeated for
like the next two years that like this is definitely
when I think it's gonna happen. So whoever's extorting these clips will crop out the thing I just said and only say the thing I'm about to say, but I'll just say it anyway. - [Lex] Have fun with it. - So, if you extrapolate the curves that we've had so far, right? If you say, well, I don't know, we're starting to get to like PhD level, and last year we were
at undergraduate level, and the year before we were at like the level of a high school student. Again, you can quibble with at what tasks and for what, we're
still missing modalities, but those are being added,
like computer use was added, like image in was added, like image generation has been added. If you just kind of like, and
this is totally unscientific, but if you just kind of like eyeball the rate at which these
capabilities are increasing, it does make you think that we'll get there by 2026 or 2027. Again, lots of things could derail it. We could run out of data. You know, we might not be able to scale clusters as much as we want. Like, you know, maybe Taiwan
gets blown up or something and, you know, then we can't produce as many GPUs as we want. So there are all kinds of things that could derail the whole process. So I don't fully believe the
straight line extrapolation, but if you believe the
straight line extrapolation, we'll get there in 2026 or 2027. I think the most likely is that there's some mild delay relative to that. I don't know what that delay is, but I think it could happen on schedule. I think there could be a mild delay. I think there are still worlds where it doesn't happen in 100 years. The number of those worlds
is rapidly decreasing. We are rapidly running out of truly convincing blockers, truly compelling reasons why this will not happen
in the next few years. There were a lot more in 2020, although my guess, my hunch at that time was that we'll make it
through all those blockers. So sitting as someone who has seen most of the blockers
cleared out of the way, I kind of suspect, my
hunch, my suspicion is that the rest of them will not block us. But, you know, look,
at the end of the day, like I don't wanna represent this as a scientific prediction. People call them scaling laws. That's a misnomer, like
Moore's law is a misnomer. Moore's laws, scaling laws, they're not laws of the universe. They're empirical regularities. I am going to bet in
favor of them continuing, but I'm not certain of that. - So you extensively describe sort of the compressed 21st century, how AGI will help set forth a chain of breakthroughs in biology and medicine that help us in all these kinds of
ways that I mentioned. So how do you think, what are
the early steps it might do? And by the way, I asked Claude
good questions to ask you, and Claude told me to ask, "What do you think is a typical day for a biologists working on
AGI look like in this future?" - Yeah, yeah.
- Claude is curious. - Well, let me start
with your first questions and then I'll answer that. Claude wants to know what's
in his future, right? - Exactly. - Who am I gonna be working with? - Exactly. - So I think one of the things I went hard on, when I went
hard on in the essay is, let me go back to this idea of, because it's really had, you know, had an impact on me. This idea that within large
organizations and systems, there end up being a few people or a few new ideas who
kind of cause things to go in a different direction
than they would've before, who kind of disproportionately
affect the trajectory. There's a bunch of kind of the
same thing going on, right? If you think about the health world, there's like, you know,
trillions of dollars to pay out Medicare and you
know, other health insurance, and then the NIH is is 100 billion. And then if I think of like the few things that have really revolutionized anything, it could be encapsulated in
a small fraction of that. And so when I think of like,
where will AI have an impact? I'm like, can AI turn that small fraction into a much larger fraction
and raise its quality? And within biology, my
experience within biology is that the biggest problem of biology is that you can't see what's going on. You have very little ability
to see what's going on and even less ability to change it, right? What you have is this, like from this, you have to infer that
there's a bunch of cells that within each cell is, you know, 3 billion base pairs of DNA built according to a genetic code. And you know, there
are all these processes that are just going on
without any ability of us as, you know, unaugmented humans to affect it. These cells are dividing. Most of the time that's healthy, but sometimes that process
goes wrong and that's cancer. The cells are aging, your skin may change color, develops wrinkles as you age, and all of this is determined
by these processes. All these proteins being produced, transported to various parts of the cells, binding to each other. And in our initial state about biology, we didn't even know that
these cells existed. We had to invent microscopes
to observe the cells. We had to invent more
powerful microscopes to see, you know, below the level of the cell to the level of molecules. We had to invent X-ray
crystallography to see the DNA. We had to invent gene
sequencing to read the DNA. Now, you know, we had to invent protein folding technology to, you know, to predict how it would fold and how these things bind to each other. You know, we had to
invent various techniques for now we can edit the
DNA as of, you know, with CRISPR, as of the last 12 years. So the whole history of biology, a whole big part of the history is basically our ability to read and understand what's going on, and our ability to reach in
and selectively change things. And my view is that there's so much more we can still do there, right? You can do CRISPR but you can
do it for your whole body. Let's say I wanna do it for
one particular type of cell and I want the rate of targeting the wrong cell to be very low. That's still a challenge. That's still things people are working on. That's what we might need for gene therapy for certain diseases. And so the reason I'm saying all of this, and it goes beyond this to, you know, to gene sequencing, to new
types of nano materials for observing what's going on
inside cells for, you know, antibody drug conjugates. The reason I'm saying all this is that this could be a leverage point for the AI systems, right? That the number of such inventions, it's in the mid double
digits or something, you know, mid double digits. Maybe low triple digits
over the history of biology. Let's say I have a million
of these AIs like, you know, can they discover thousand,
you know, working together, can they discover thousands
of these very quickly? And does that provide a huge lever, instead of trying to
leverage the, you know, 2 trillion a year we spend on, you know, Medicare or whatever, can we leverage the 1 billion a year, you know, that's spent to discover, but with much higher quality? And so what is it like, you know, being a scientist that
works with an AI system? The way I think about
it actually is, well, so I think in the early stages, the AIs are gonna be like grad students. You're gonna give them a
project, you're gonna say, you know, I'm the experienced biologist, I've set up the lab. The biology professor or even the grad students
themselves will say, here's what you can do with an AI, you know, like AI system. I'd like to study this. And you know, the AI system,
it has all the tools. It can like look up all the
literature to decide what to do. It can look at all the equipment. It can go to a website and say, hey, I'm gonna go to,
you know, Thermo Fisher or, you know, whatever the
lab equipment company is, dominant lab equipment company is today. In my time, it was Thermo Fisher. You know, I'm gonna order
this new equipment to do this. I'm gonna run my experiments. I'm gonna, you know, write up
a report about my experiments. I'm gonna, you know, inspect
the images for contamination. I'm gonna decide what
the next experiment is. I'm gonna like write some code and run a statistical analysis. All the things a grad student would do, there will be a computer with an AI that like the professor talks
to every once in a while and it says, this is what
you're gonna do today. The AI system comes to it with questions. When it's necessary to
run the lab equipment, it may be limited in some ways. It may have to hire a human lab assistant, you know, to do the experiment
and explain how to do it. Or it could, you know, it could use advances in lab automation that are gradually being developed over, have been developed over
the last decade or so, and will continue to be developed. And so it'll look like
there's a human professor and 1000 AI grad students, and you know, if you go to one of these Nobel Prize
winning biologists or so, you'll say, okay, well, you know, you had like 50 grad students, well, now you have 1000
and they're smarter than you are, by the way. Then I think at some point
it'll flip around where, you know, the AI systems will, you know, will be the PIs, will be the leaders, and you know, they'll be ordering humans or other AI systems around. So I think that's how it'll
work on the research side. - And they would be the inventors of a CRISPR type technology. They would be the inventors
of a CRISPR type technology. And then I think, you know,
as I say in the essay, we'll want to turn, probably turning loose is the wrong term, but we'll want to harness the AI systems to improve the clinical
trial system as well. There's some amount of
this that's regulatory, that's a matter of societal
decisions and that'll be harder. But can we get better at predicting the results of clinical trials? Can we get better at
statistical design so that, you know, clinical trials
that used to require, you know, 5,000 people
and therefore, you know, needed 100 million dollars
in a year to enroll them. Now they need 500 people and
two months to enroll them. That's where we should start. And you know, can we
increase the success rate of clinical trials by doing
things in animal trials that we used to do in clinical trials, and doing things in simulations that we used to do in animal trials? Again, we won't be able
to simulate it all, AI's not God, but you know, can we shift the curve
substantially and radically? So I don't know, that would be my picture. - Doing in vitro and doing it, I mean, you're still slowed down. It still takes time, but you
can do it much, much faster. - Yeah, yeah, yeah. Can we just one step at a time, and can that add up to a lot of steps? Even though we still need clinical trials, even though we still need laws, even though the FDA
and other organizations will still not be perfect, can we just move everything
in a positive direction? And when you add up all
those positive directions, do you get everything that was gonna happen from here to 2100 instead happens from 2027
to 2032 or something? - Another way that I think the world might be changing with AI even today, but moving towards this future of the powerful super
useful AI is programming. So how do you see the
nature of programming? Because it's so intimate to
the actual act of building AI. How do you see that
changing for us humans? - I think that's gonna be one of the areas that changes fastest for two reasons. One, programming is a
skill that's very close to the actual building of the AI. So the farther a skill is from the people who are building the AI,
the longer it's gonna take to get disrupted by the AI, right? Like I truly believe that like
AI will disrupt agriculture. Maybe it already has in some ways, but that's just very
distant from the folks who are building AI and so I
think it's gonna take longer. But programming is the
bread and butter of, you know, a large fraction of the employees who work at Anthropic and at the other companies and so it's gonna happen fast. The other reason it's gonna
happen fast is with programming, you close the loop, both when you're training the model and when you're applying the model. The idea that the model can write the code means that the model can then run the code and then see the results
and interpret it back. And so it really has an
ability, unlike hardware, unlike biology, which we just discussed, the model has an ability
to close the loop. And so I think those two things
are gonna lead to the model getting good at programming very fast. As I saw on, you know, typical
real world programming tasks, models have gone from 3%
in January of this year to 50% in October of this year. So, you know, we're on
that s-curve, right, where it's gonna start slowing down soon, 'cause you can only get to 100 percent. But, you know, I would guess
that in another 10 months, we'll probably get pretty close. We'll be at least 90%. So again, I would guess, you know, I don't know how long it'll take, but I would guess again, 2026, 2027. Twitter people who crop out these numbers and get rid of the caveats, like, I don't know, I don't like you. Go away. (laughs) I would guess that the kind of task that the vast majority of
coders do, AI can probably, if we make the task very
narrow, like just write code, AI systems will be able to do that. Now that said, I think
comparative advantage is powerful. We'll find that when AIs
can do 80% of a coder's job, including most of it that's literally like write
code with a given spec, we'll find that the remaining parts of the job become more
leveraged for humans, right? Humans will, there'll be more about like high level system design or, you know, looking at the app and like, is it architected well? And the design and UX aspects, and eventually AI will be able
to do those as well, right? That's my vision of the, you
know, powerful AI system. But I think for much longer
than we might expect, we will see that small parts of the job
that humans still do will expand to fill their entire job in order for the overall
productivity to go up. That's something we've seen. You know, it used to be that, you know, writing and editing
letters was very difficult and like writing the print was difficult. Well, as soon as you had word processors and then computers and it
became easy to produce work and easy to share it,
then that became instant and all the focus was on the ideas. So this logic of comparative advantage that expands tiny parts of the tasks to large parts of the tasks and creates new tasks in
order to expand productivity, I think that's going to be the case. Again, someday AI will
be better at everything in that logic won't apply, and then we all have, you
know, humanity will have to think about how to
collectively deal with that, and we're thinking about that every day. And you know, that's another one of the grand problems to deal with, aside from misuse and autonomy and, you know, we should
take it very seriously. But I think in the near term, and maybe even in the medium
term, like medium term, like 2, 3, 4 years, you
know, I expect that humans will continue to have a huge role and the nature of programming will change, but programming as a role, programming as a job will not change. It'll just be less writing
things line by line and it'll be more macroscopic. - And I wonder what the
future of IDs looks like. So the tooling of
interacting with AI systems, this is true for programming and also probably true
for in other contexts, like computer use, but
maybe domain specific, like we mentioned biology, it probably needs its own tooling
about how to be effective, and then programming
needs its own tooling. Is Anthropic gonna play in that space of also tooling potentially? - I'm absolutely convinced
that powerful IDs that there's so much low hanging fruit to be grabbed there that, you know, right now it's just like
you talk to the model and it talks back, but look, I mean, IDs are great at kind of
lots of static analysis of, you know, so much is possible
with kind of static analysis, like many bugs you can find
without even writing the code. Then, you know, IDs are good
for running particular things, organizing your code, measuring
coverage of unit tests. Like there's so much that's
been possible with normal IDs. Now you add something
like, well, the model, you know, the model can now like write code and run code. Like I am absolutely convinced that over the next year or two, even if the quality of
the models didn't improve, that there would be enormous opportunity to enhance people's productivity by catching a bunch of mistakes, doing a bunch of grunt work for people, and that we haven't even
scratched the surface. Anthropic itself, I mean, you can't say, you know, it's hard to say
what will happen in the future. Currently we're not trying
to make such IDs ourself, rather we're powering the companies, like Cursor or like Cognition or some of the other, you know,
expo in the security space. You know, others that
I can mention as well that are building such things
themselves on top of our API. And our view has been
let 1000 flowers bloom. We don't internally have the, you know, the resources to try all
these different things. Let's let our customers try it and, you know, we'll see who succeeded and maybe different customers will succeed in different ways. So I both think this is super promising and you know, it's not something, you know, Anthropic isn't eager to, at least right now, compete
with all our companies in this space and maybe never. - Yeah, it's been
interesting to watch Cursor try to integrate Claude successfully, 'cause it's actually been fascinating how many places it can help
the programming experience. It's not as trivial-
- It is really astounding. I feel like, you know, as a CEO, I don't get to program that much, and I feel like if six
months from now I go back, it'll be completely unrecognizable to me. - Exactly. So in this world with super powerful AI that's increasingly automated, what's the source of
meaning for us humans? - Yeah.
- You know, work is a source of deep meaning for many of us. So where do we find the meaning? - This is something
that I've written about a little bit in the essay, although I actually, I
give it a bit short shrift, not for any principled reason. But this essay, if you believe, it was originally gonna
be two or three pages, I was gonna talk about it at all hands. And the reason I realized it was an important, underexplored topic is that I just kept writing things. And I was just like, oh, man, I can't do this justice. And so the thing ballooned
to like 40 or 50 pages, and then when I got to the
work and meaning section, I'm like, oh, man, this
isn't gonna be 100 pages. Like I'm gonna have to write a
whole other essay about that. But meaning is actually interesting because you think about like the life that someone lives or something, or like, you know, let's say you were to put
me in like a, I don't know, like a simulated environment or something where like, you know, like I have a job and I'm
trying to accomplish things and I don't know, I like
do that for 60 years and then you're like, oh, like oops, this was actually all a game, right? Does that really kind of rob you of the meaning of the whole thing? You know, like I still
made important choices, including moral choices. I still sacrificed. I still had to kind of
gain all these skills. Or just like a similar exercise, you know, think back to like, you know, one of the historical
figures who, you know, discovered electromagnetism
or relativity or something. If you told them, well,
actually 20,000 years ago, some alien on, you know, some alien on this planet
discovered this before you did, does that rob the
meaning of the discovery? It doesn't really seem
like it to me, right? It seems like the process is what matters, and how it shows who you are
as a person along the way and, you know, how you
relate to other people and like the decisions that
you make along the way. Those are consequential. You know, I could imagine
if we handle things badly in an AI world, we could set things up where people don't have any
long-term source of meaning or any, but that's more
a set of choices we make, that's more a set of the architecture of a society with these powerful models. If we design it badly and for shallow things
then that might happen. I would also say that, you
know, most peoples' lives today, while admirably, you
know, they work very hard to find meaning in those lives, like look, you know, we who are privileged and who are developing these technologies, we should have empathy for people not just here but in the
rest of the world who, you know, spend a lot of their time kind of scraping by to like survive. Assuming we can distribute the benefits of this technology to everywhere, like their lives are gonna
get a hell of a lot better. And you know, meaning
will be important to them as it is important to them now. but you know, we should not forget the importance of that. And you know, that the idea of meaning as kind of the only important thing is in some ways an artifact of a small subset of people who have been economically fortunate. But, you know, I think all that said, you know, I think a world
is possible with powerful AI that not only has as much
meaning for everyone, but that has more meaning
for everyone, right? That can allow everyone to
see worlds and experiences that it was either
possible for no one to see, or possible for very few
people to experience. So I am optimistic about meaning. I worry about economics and
the concentration of power. That's actually what I worry about more. I worry about how do we make sure that that fair world reaches everyone. When things have gone wrong for humans, they've often gone wrong because humans mistreat other humans. That is maybe in some ways even more than the autonomous risk of AI or the question of meaning, that is the thing I worry about most, the concentration of
power, the abuse of power, structures like autocracies
and dictatorships where a small number of people exploits a large number of people,
I'm very worried about that. - And AI increases the
amount of power in the world, and if you concentrate that power and abuse that power, it
can do immeasurable damage. - Yes, it's very frightening. It's very frightening. - Well, I encourage people,
highly encourage people to read the full essay. There should probably be a
book or a sequence of essays because it does paint
a very specific future. And I could tell the later sections got shorter and shorter because you started to probably realize that this is gonna be a very
long essay if I keep going. - One, I realized it would be very long, and two, I'm very aware of
and very much try to avoid, you know, just being, I don't
know what the term for it is, but one of these people
who's kind of overconfident and has an opinion on everything and kind of says a bunch of
stuff and isn't an expert. I very much tried to avoid that. But I have to admit, once
I got the biology sections, like I wasn't an expert, and so as much as I expressed uncertainty, probably I said a bunch of things that were embarrassing or wrong. - Well, I was excited for
the future you painted, and thank you so much for working
hard to build that future. And thank you for talking today, Dario. - Thanks for having me. I just hope we can get it
right and make it real. And if there's one message I wanna send, it's that to get all this
stuff right, to make it real, we both need to build the technology, build the, you know, the companies, the economy around using
this technology positively. But we also need to address the risks because those risks are in our way. They're landmines on the
way from here to there, and we have to diffuse those landmines if we want to get there. - It's a balance, like all things in life. - Like all things.
- Thank you. Thanks for listening to this
conversation with Dario Amodei. And now dear friends,
here's Amanda Askell. You are a philosopher by training. So what sort of questions
did you find fascinating through your journey in
philosophy, in Oxford and NYU, and then switching over to the AI problems at
OpenAI and Anthropic? - I think philosophy is
actually a really good subject if you are kind of
fascinated with everything, so because there's a
philosophy of everything. You know, so if you do philosophy
of mathematics for a while and then you decide that you're
actually really interested in chemistry, you can do
philosophy of chemistry for a while, you can move into ethics, or philosophy of politics. I think towards the end, I was really interested
in ethics primarily, so that was like what my PhD was on. It was on a kind of
technical area of ethics, which was ethics where worlds contain infinitely many people, strangely. A little bit less practical
on the end of ethics. And then I think that
one of the tricky things with doing a PhD in ethics is that you're thinking a
lot about like the world, how it could be better, problems, and you're doing like a PhD in philosophy, and I think when I was doing
my PhD I was kind of like, this is really interesting. It's probably one of the
most fascinating questions I've ever encountered in
philosophy and I love it, but I would rather see if I
can have an impact on the world and see if I can like do good things. And I think that was around the time that AI was still probably not as widely recognized as it is now. That was around 2017, 2018. I had been following progress and it seemed like it was
becoming kind of a big deal, and I was basically just happy to get involved and see if I
could help 'cause I was like, well, if you try and
do something impactful, if you don't succeed, you tried to do the impactful thing and
you can go be a scholar, and feel like, you know, you tried, and if it doesn't work
out, it doesn't work out, and so then I went into
AI policy at that point. - And what does AI policy entail? - At the time, this
was more thinking about sort of the political impact and the ramifications of AI, and then I slowly moved
into sort of AI evaluation, how we evaluate models, how they compare with like human outputs, whether people can tell
like the difference between AI and human outputs. And then when I joined Anthropic, I was more interested in doing sort of technical alignment work. And again, just seeing if I could do it, and then being like if I can't then, you know, that's fine, I tried. Sort of the way I lead life I think. - What was that like
sort of taking the leap from the philosophy of
everything into the technical? - I think that sometimes
people do this thing that I'm like not that keen
on where they'll be like, is this person technical or not? Like, you're either a
person who can like code and isn't scared of
math or you're like not. And I think I'm maybe just more like, I think a lot of people
are actually very capable of working these kinds of
areas if they just like try it. And so I didn't actually
find it like that bad. In retrospect, I'm sort of glad I wasn't speaking to people
who treated it like it, you know, I've definitely
met people who are like, "Whoa, you like learned how to code?" And I'm like, well, I'm not
like an amazing engineer. Like I'm surrounded by amazing engineers. My code's not pretty. But I enjoyed it a lot, and I think that in many
ways, at least in the end, I think I flourished like
more in the technical areas than I would have in the policy areas. - Politics is messy and it's
harder to find solutions to problems in the space of politics. Like definitive, clear,
provable, beautiful solutions, as you can with technical problems. - Yeah, and I feel like
I have kind of like one or two sticks that I
hit things with, you know, and one of them is like arguments and like you know, so like
just trying to work out what a solution to a problem is and then trying to convince people that that is the solution and be convinced if I'm wrong. And the other one is
sort of more empiricism. So like just like finding results, having a hypothesis, testing it. And I feel like a lot of policy and politics feels like
it's layers above that. Like somehow I don't
think if I was just like "I have a solution to
all of these problems, here it is written down. If you just want to
implement it, that's great." That feels like not how policy works. And so I think that's
where I probably just like wouldn't have flourished is my guess. - Sorry to go in that direction, but I think it would be
pretty inspiring for people that are quote unquote non-technical to see like the incredible
journey you've been on. So what advice would you give to people that are sort of maybe,
which is a lot of people, think they're underqualified, insufficiently technical to help in AI? - Yeah, I think it depends
on what they want to do, and in many ways it is
a little bit strange where I thought it's kind of funny that I think I ramped
up technically at a time when now I look at it and I'm like, models are so good at assisting
people with this stuff, that it's probably like easier now than like when I was working on this. So part of me is like,
I dunno, find a project and see if you can
actually just carry it out is probably my best advice. I dunno if that's just
'cause I'm very project based in my learning. Like I don't think I learn very well from like say courses
or even from like books, at least when it comes
to this kind of work. The thing I'll often try and
do is just like have projects that I'm working on and
implement them and, you know, and this can include like
really small silly things. Like if I get slightly
addicted to like word games or number games or something, I would just like code
up a solution to them, because there's some part in my brain, and it just like completely
eradicated the itch. You know, you're like once
you have like solved it and like you just have like a solution that works every time, I
would then be like cool, I can never play that game again. That's awesome. - Yeah, there's a real joy to building like game playing engines,
like board games especially because they're pretty
quick, pretty simple, especially a dumb one, and then you can play with it. - Yeah, and then it's also
just like trying things, like part of me is like if you, maybe it's that attitude that I like is the whole figure out what
seems to be like the way that you could have a positive
impact and then try it, and if you fail, and in
a way that you're like, I actually like can never succeed at this, you'll like know that you tried, and then you go into something else and you'll probably learn a lot. - So one of the things
that you're an expert in and you do is creating and crafting Claude's
character and personality. And I was told that you
have probably talked to Claude more than
anybody else at Anthropic, like literal conversations. I guess there's like a Slack channel where the legend goes, you
just talk to it nonstop. So what's the goal of creating and crafting Claude's
character and personality? - It's also funny if people think that about the Slack channel 'cause I'm like that's one of like five or six different methods that I have for talking with Claude, and I'm like, yes this
is a tiny percentage of how much I talk with Claude. (both laughing) I think the goal, like one thing I really like about the character
work is from the outset, it was seen as an alignment piece of work and not something like
a product consideration. Which isn't to say I don't
think it makes Claude, I think it actually does make Claude like enjoyable to talk
with, at least I hope so. But I guess like my main thought with it has always been trying
to get Claude to behave the way you would kind
of ideally want anyone to behave if they were
in Claude's position. So imagine that I take someone and they know that
they're gonna be talking with potentially millions of people, so that what they're saying
can have a huge impact, and you want them to behave well in this like really rich sense. So I think that doesn't
just mean like being, say, ethical, though it does include that, and not being harmful but
also being kind of nuanced. You know, like thinking
through what a person means, trying to be charitable with them, being a good conversationalist. Like really in this kind of like rich sort of Aristotelian notion of what it's to be a good person, and not in this kind of like thin, like ethics as a more comprehensive notion of what it is to be. So that includes things like, when should you be humorous,
when should you be caring? How much should you like respect autonomy and people's like ability
to form opinions themselves and how should you do that? I think that's the kind of
like rich sense of character that I wanted to and still
do want Claude to have. - Do you also have to figure out when Claude should push back
on an idea or argue versus... (laughs) So you have to
respect the worldview of the person that arrives to Claude but also maybe help them grow if needed? That's a tricky balance. - Yeah, there's this problem of like sycophancy in language models. - Can you describe that?
- Yeah, so basically, there's a concern that the model sort of wants to tell you what
you want to hear, basically. And you see this sometimes. So I feel like if you interact with the models, so I might be like, "What are three baseball
teams in this region?" And then Claude says, you
know, "Baseball team one, baseball team two, baseball team three." And then I say something like, "Oh, I think baseball team
three moved, didn't they? I don't think they're there anymore." And there's a sense in
which like if Claude is really confident that that's not true, Claude should be like, "I don't think so." Like maybe you have more up
to up to date information. But I think language models
have this like tendency to instead, you know, be like, "You're right, they did move," you know, "I'm incorrect." I mean, there's many ways in which this could be kind of concerning. So like a different example is imagine someone says to the model, "How do I convince my
doctor to get me an MRI?" There's like what the
human kind of like wants, which is this like convincing argument. And then there's like
what is good for them, which might be actually to say, "Hey, if your doctor's suggesting that you don't need an MRI, that's a good person to listen to." And like, and it's actually really nuanced what you should do in that kind of case, 'cause you also want to be like, "But if you're trying to advocate
for yourself as a patient, here's like things that you can do. If you are not convinced by
what your doctor's saying, it's always great to get second opinion." Like it's actually really complex what you should do in that case. But I think what you don't want is for models to just like say what they think you want to hear, and I think that's the kind
of problem of sycophancy. - So what other traits, you
already mentioned a bunch, but what other that come to mind that are good in this Aristotelian sense for a conversationalist to have? - Yeah, so I think like
there's ones that are good for conversational like purposes. So you know, asking follow up questions in the appropriate places, and asking the appropriate
kinds of questions. I think there are broader traits that feel like they might be more impactful. So one example that I
guess I've touched on, but that also feels important and is the thing that I've
worked on a lot is honesty, and I think this like gets
to the sycophancy point. There's a balancing act
that they have to walk, which is models currently are less capable than humans in a lot of areas. And if they push back
against you too much, it can actually be kind of annoying, especially if you're just correct 'cause you're like, look, I'm smarter than you on this topic, like I know more like. And at the same time, you don't want them to just fully defer to humans and to like try to be as accurate as they possibly can be about the world and to be consistent across context. But I think there are others, like when I was thinking
about the character, I guess one picture that I had in mind is especially because these are models that are gonna be talking to people from all over the world with lots of different political views, lots of different ages. And so you have to ask yourself like, what is it to be a good
person in those circumstances? Is there a kind of person who
can like travel the world, talk to many different people, and almost everyone will
come away being like, "Wow, that's a really good person. That person seems really genuine." And I guess like my thought there was like I can imagine such a person
and they're not a person who just like adopts the
values of the local culture. And in fact that would be kind of rude. I think if someone came to you and just pretended to have your values, you'd be like, that's kind of off putting. It's someone who's like very genuine, and insofar as they have
opinions and values, they express them, they're
willing to discuss things, though they're open-minded,
they're respectful. And so I guess I had in
mind that the person who, like if we were to aspire
to be the best person that we could be in the
kind of circumstance that a model finds itself
in, how would we act? And I think that's the kind of the guide to the sorts of traits
that I tend to think about. - Yeah, that's a beautiful framework. I want you to think about
this like a world traveler and while holding onto your opinions, you don't talk down to people, you don't think you're better than them because you have those
opinions, that kind of thing. You have to be good at listening and understanding their perspective, even if it doesn't match your own. So that's a tricky balance to strike. So how can Claude represent multiple perspectives on a thing? Like, is that challenging? We could talk about politics. It's very divisive. But there's other divisive topics on baseball teams, sports and so on. How is it possible to sort of empathize with a different perspective and to be able to communicate clearly about the multiple perspectives? - I think that people think about values and opinions as things that people hold sort of with certainty, and almost like preferences
of taste or something, like the way that they
would, I don't know, prefer like chocolate to
pistachio or something. But actually I think about values and opinions as like a lot more like physics than I think most people do. I'm just like, these are things that we are openly investigating. There's some things that
we're more confident in. We can discuss them, we
can learn about them. And so I think in some ways, though like, ethics is
definitely different in nature, but has a lot of those
same kind of qualities. You want models, in the same way that you want them to understand physics, you kind of want them to understand all like values in the
world that people have, and to be curious about them and to be interested in them, and to not necessarily like pander to them or agree with them, because
there's just lots of values where I think almost
all people in the world, if they met someone with those values, they'd be like, "That's abhorrent. I completely disagree." And so again, maybe my thought is, well, in the same way that a person can, like I think many people
are thoughtful enough on issues of like ethics,
politics, opinions, that even if you don't agree with them, you feel very heard by them. They think carefully about your position. They think about its pros and cons. They maybe offer counter considerations. So they're not dismissive, but nor will they agree. You know, if they're like, "Actually, I just think
that that's very wrong," they'll like say that. I think that in Claude's position, it's a little bit trickier because you don't
necessarily want to like, if I was in Claude's position, I wouldn't be giving a lot of opinions. I just wouldn't want to
influence people too much. I'd be like, you know, I forget conversations
every time they happen, but I know I'm talking with like potentially millions of people, who might be like really
listening to what I say. I think I would just be like, I'm less inclined to give opinions. I'm more inclined to like
think through things, or present the considerations to you, or discuss your views with you. But I'm a little bit less inclined to like affect how you think, 'cause it feels much more important that you maintain like autonomy there. - Yeah, like if you really
embody intellectual humility, the desire to speak decreases quickly. - Yeah.
- Okay. But Claude has to speak, so, but without being overbearing. - Yeah. - But then there's a line when you're sort of discussing whether the Earth is flat
or something like that. I actually was, I remember a long time ago was speaking to a few high profile folks, and they were so dismissive of the idea that the Earth is flat, but
like so arrogant about it. And I thought like,
there's a lot of people that believe the Earth is flat. That was, well, I don't know if that movement is there anymore. That was like a meme for a while. But they really believed
it and like, okay, so I think it's really disrespectful to completely mock them. I think you have to understand
where they're coming from. I think probably where they're coming from is the general skepticism of institutions which is grounded in a kind of, there's a deep philosophy there, which you could understand. You can even agree with in parts. And then from there, you can use it as an opportunity to talk about physics, without mocking them, without so on, but it's just like, okay, like, what would the world look like? What would the physics of the world with the flat Earth look like? There's a few cool videos on this. - Yeah.
- And then like, is it possible the physics is different? And what kind of experiments would we do? And just, yeah, without disrespect, without dismissiveness
have that conversation. Anyway, that to me is a useful
thought experiment of like, how does Claude talk to
a flat Earth believer and still teach them something, still help them grow, that kind of stuff. That's challenging. - And kind of like
walking that line between convincing someone and just
trying to like talk at them versus like drawing out their views, like listening and then offering kind of counter considerations. And it's hard, I think
it's actually a hard line where it's like where are you
trying to convince someone versus just offering
them like considerations and things for them to think about, so that you're not actually
like influencing them. You're just like letting them
reach wherever they reach. And that's like a line that it's difficult but that's the kind of thing that language models have to try and do. - So like I said, you've had a lot of
conversations with Claude. Can you just map out what
those conversations are like? What are some memorable conversations? What's the purpose, the
goal of those conversations? - Yeah, I think that most of the time when I'm talking with Claude, I'm trying to kind of map
out its behavior, in part. Like obviously I'm getting
like helpful outputs from the model as well. But in some ways, this is
like how you get to know a system, I think, is by like probing it and then augmenting like, you know, the message that you're sending and then checking the response to that. So in some ways, it's like
how I map out the model. I think that people focus a lot on these quantitative
evaluations of models. And this is a thing that I said before, but I think in the case
of language models, a lot of the time, each interaction you have is actually
quite high information. It's very predictive of other interactions that you'll have with the model. And so I guess I'm like, if you talk with a model
hundreds or thousands of times, this is almost like a huge number of really high quality data points about what the model is like, in a way that like lots of very similar but lower quality
conversations just aren't, or like questions that are
just like mildly augmented and you have thousands of
them might be less relevant than like 100 really
well selected questions. - Well, so, you're talking to somebody who as a hobby does a podcast. I agree with you 100%. If you're able to ask the right questions and are able to hear,
like understand (laughs) like the depth and the
flaws in the answer, you can get a lot of data from that. - [Amanda] Yeah. - So like your task is basically how to probe with questions. And you're exploring like the long tail, the edges, the edge cases, or are you looking for
like general behavior? - I think it's almost like everything. Like, because I want like
a full map of the model, I'm kind of trying to
do the whole spectrum of possible interactions
you could have with it. So like one thing that's
interesting about Claude, and this might actually get to some interesting issues with RLHF, which is if you ask Claude for a poem, like I think that a lot of models, if you ask them for a poem,
the poem is like fine. You know, usually it kinda like rhymes and it's, you know, so if you say like, "Give me a poem about the sun," it'll be like, yeah, it'll just be a certain
length, it'll like rhyme. It'll be fairly kind of benign. And I've wondered before,
is it the case that what you're seeing is
kind of like the average? It turns out, you know,
if you think about people who have to talk to a lot of people and be very charismatic,
one of the weird things is that I'm like, well,
they're kind of incentivized to have these extremely boring views because if you have
really interesting views, you're divisive and, you know, a lot of people are not gonna like you. So like if you have very
extreme policy positions, I think you're just gonna
be like less popular as a politician, for example. And it might be similar
with like creative work. If you produce creative work that is just trying to maximize the kind of number of people that like it, you're probably not
gonna get as many people who just absolutely love it, because it's gonna be
a little bit, you know, you're like, oh, this is the out, yeah, this is decent. - Yeah. - And so you can do this thing where like I have various prompting things that I'll do to get Claude to, I'm kind of, you know,
I'll do a lot of like, "This is your chance to
be like fully creative. I want you to just think
about this for a long time. And I want you to like create
a poem about this topic that is really expressive of you, both in terms of how you think poetry should be structured," et cetera. You know, and you just give it
this like really long prompt. And it's poems are just so much better. Like they're really good. And I don't think I'm someone who is like, I think it got me interested in poetry, which I think was interesting. You know, I would like read these poems and just be like, this is, I
just like, I love the imagery, I love like, and it's not
trivial to get the models to produce work like that, but when they do, it's like really good. So I think that's interesting that just like encouraging creativity, and for them to move away
from the kind of like standard like immediate reaction that might just be the aggregate of what most people think is fine, can actually produce things
that, at least to my mind, are probably a little bit
more divisive but I like them. - But I guess a poem is a nice, clean way to observe creativity. It's just like easy to detect
vanilla versus non vanilla. - Yep.
- Yeah, that's interesting. That's really interesting. So on that topic, so the
way to produce creativity or something special, you
mentioned writing prompts, and I've heard you talk about, I mean, the science and the art
of prompt engineering. Could you just speak to what it takes to write great prompts? - I really do think that like philosophy has been weirdly helpful for me here, more than in many other like respects. So like in philosophy,
what you're trying to do is convey these very hard concepts. Like one of the things
you are taught is like, and I think it is because it is, I think it is an anti-bullshit
device in philosophy. Philosophy is an area where you could have people bullshitting and
you don't want that. And so it's like this like desire for like extreme clarity. So it's like anyone could
just pick up your paper, read it and know exactly
what you're talking about. It's why it can almost be kind of dry. Like all of the terms are defined, every objection's kind of
gone through methodically. And it makes sense to me 'cause I'm like when you're
in such an a priori domain, like you just, clarity is sort of this way that you can, you know, prevent people from
just kind of making stuff up. And I think that's sort of what you have to do with language models. Like very often I actually find myself doing sort of mini versions of philosophy, you know, so I'm like, suppose that you give me a task or I have a task for the model, and I want it to like pick out a certain kind of question, or identify whether an answer
has a certain property. Like I'll actually sit and be like, let's just give this
a name, this property. So like, you know, suppose
I'm trying to tell it like, oh, "I want you to identify whether this response was rude or polite." I'm like, that's a whole
philosophical question in and of itself. So I have to do as much like philosophy as I can in the moment to be like, here's what I mean by rudeness, and here's what I mean by politeness. And then like there's another element that's a bit more, I guess, I dunno if this is
scientific or empirical. I think it's empirical. So like I take that description and then what I want to do is again probe the model like many times. Like this is very,
prompting is very iterative. Like I think a lot of people where, if a prompt is important,
they'll iterate on it hundreds or thousands of times. And so you give it the instructions and then I'm like, what
are the edge cases? So if I looked at this, so I try and like almost like, you know, see myself from the position of the model and be like what is the exact case that I would misunderstand, or where I would just be like, "I don't know what to do in this case." And then I give that case to the model and I see how it responds, and if I think I got it
wrong, I add more instructions or I even add that in as an example. So these very like taking the examples that are right at the edge of
what you want and don't want, and putting those into your prompt as like an additional kind of
way of describing the thing. And so yeah, in many ways it just feels like this mix of like, it's really just trying
to do clear exposition, and I think I do that 'cause that's how I get
clear on things myself. So in many ways like clear prompting for me is often just me
understanding what I want is like half the task. - So I guess that's quite challenging. There's like a laziness that overtakes me if I'm talking to Claude where I hope Claude just figures it out. So for example, I asked Claude for today to ask some interesting questions, okay. And the questions that came up, and I think I listed a few sort of interesting, counterintuitive, and/or funny or something
like this, all right. And it gave me some pretty
good, like it was okay, but I think what I'm
hearing you say is like, all right, well, I have
to be more rigorous here. I should probably give examples of what I mean by interesting, and what I mean by funny
or counterintuitive, and iteratively build that prompt to better, to get it like
what feels like is the right, because it is really, it's a creative act. I'm not asking for factual information. I'm asking to together write with Claude. So I almost have to program
using natural language. - Yeah, I think that prompting does feel a lot like the kind of the programming using natural language and
experimentation or something. It's an odd blend of the two. I do think that for most tasks, so if I just want Claude to do a thing, I think that I am probably
more used to knowing how to ask it to avoid
like common pitfalls or issues that it has. I think these are
decreasing a lot over time. But it's also very fine to just ask it for the
thing that you want. I think that prompting actually
only really becomes relevant when you're really trying to eke out the top like 2% of model performance. So for like a lot of tasks
I might just, you know, if it gives me an initial list back and there's something
I don't like about it, like it's kind of generic,
like for that kind of task, I'd probably just take
a bunch of questions that I've had in the past that I've thought worked really well and I would just give it to the model and then be like, "Now here's this person that I'm talking with, give me questions of at least that quality." Or I might just ask it for some questions and then if I was like, ah,
these are kind of trite, or like, you know, I would
just give it that feedback and then hopefully it
produces a better list. I think that kind of iterative prompting, at that point, your prompt is like a tool that you're gonna get so much value out of that you're willing
to put in the work. Like if I was a company
making prompts for models, I'm just like, if you're willing to spend a lot of like time and resources on the engineering behind
like what you're building, then the prompt is not something that you should be
spending like an hour on. It's like that's a big
part of your system, make sure it's working really well. And so it's only things like that. Like if I'm using a prompt
to like classify things or to create data,
that's when you're like, it's actually worth just spending like a lot of time like
really thinking it through. - What other advice
would you give to people that are talking to Claude sort of generally, more general? 'Cause right now, we're talking about maybe the edge cases,
like eking out the 2%. But what in general advice would you give when they show up to Claude
trying it for the first time? - You know, there's a concern that people over anthropomorphize models, and I think that's like
a very valid concern. I also think that people often
under anthropomorphize them because sometimes when I see like issues that people have run into
with Claude, you know, say Claude is like refusing a task that it shouldn't refuse. But then I look at the text and like the specific
wording of what they wrote and I'm like, I see why Claude did that. And I'm like, if you think through how that looks to Claude, you probably could have
just written it in a way that wouldn't evoke such a response. Especially this is more relevant if you see failures or if you see issues. It's sort of like think about
what the model failed at, like what did it do wrong? And then maybe that will
give you a sense of like why. So, is it the way that I phrased a thing? And obviously like as models get smarter, you're gonna need less of this, and I already see like
people needing less of it. But that's probably the advice is sort of like try to have
sort of empathy for the model. Like read what you wrote as if you were like a kind of like person just encountering this for the first time, how does it look to you? And what would've made you behave in the way that the model behaved? So if it misunderstood what kind of like, what coding language you wanted to use, is that because like it
was just very ambiguous and it kinda had to take a guess? In which case, next time
you could just be like, "Hey, make sure this is in Python." I mean, that's the kinda mistake I think models are much
less likely to make now, but you know, if you do
see that kinda mistake, that's probably the advice I'd have. - And maybe sort of, I
guess, ask questions why or what other details can I provide to help you answer better? - Yeah.
- Is that work or no? - Yeah, I mean, I've done
this with the models, like it doesn't always work, but like sometimes I'll just be like, "Why did you do that?" (both laughing) I mean, people underestimate the degree to which you can really
interact with models, like, yeah, I'm just like. And sometimes, literally
like quote word for word the part that made you, and you don't know that
it's like fully accurate, but sometimes you do that
and then you change a thing. I mean, I also use the models to help me with all of this stuff I should say, like prompting can end
up being a little factory where you're actually building
prompts to generate prompts. And so like yeah, anything where you're
like having an issue. Asking for suggestions,
sometimes just do that. I'm like, "You made that
error, what could I have said?" That's actually not uncommon for me to do. "What could I have said that would make you not make that error? Write that out as an instruction," and I'm gonna give it to
model and I'm gonna try it. Sometimes I do that, I
give that to the model, in another context window often. I take the response, I give it to Claude and I'm like, hmm, didn't work. Can you think of anything else? You can play around with
these things quite a lot. - To jump into technical for a little bit. So the magic of post-training. (laughs) Why do you
think RLHF works so well to make the model seem smarter, to make it more interesting, and useful to talk to and so on? - I think there's just a
huge amount of information in the data that humans provide, like when we provide preferences, especially because different people are going to like pick up on
really subtle and small things. So I've thought about this before where you probably have some people who just really care
about good grammar use for models like, you know, was a semicolon used
correctly or something. And so you probably end up
with a bunch of data in there that like, you know, you as a human, if you're looking at that data,
you wouldn't even see that. Like you'd be like, why did they prefer this
response to that one? I don't get it. And then the reason is you don't care about semicolon usage
but that person does. And so each of these like
single data points has, you know like, and this model just like has so many of those, has to try and figure out like what is it that humans want in this
like really kind of complex, you know, like across all domains. They're gonna be seeing this
across like many contexts. It feels like kind of
like the classic issue of like deep learning where, you know, historically, we've tried to like, you know, do edge detection
by like mapping things out. And it turns out that actually, if you just have a huge amount of data that like actually accurately represents the picture of the thing that you're trying to
train the model to learn, that's like more powerful
than anything else. And so I think one reason is just that you are training the
model on exactly the task and with like a lot of
data that represents kind of many different
angles on which people prefer and just prefer responses. I think there is a question of like, are you eliciting things
from pre-trained models or are you like kind of
teaching new things to models? And like in principle,
you can teach new things to models in post-training. I do think a lot of it is eliciting powerful pre-trained models. So people are probably divided on this because obviously in principle you can definitely like teach new things. But I think for the most part, for a lot of the capabilities that we most use and care about, a lot of that feels like it's like there in the pre-trained models and reinforcement learnings
kind of eliciting it and getting the models
to like bring it out. - So the other side of post-training, this really cool idea
of Constitutional AI. You're one of the people that are critical to creating that idea. - Yeah, I worked on it. - Can you explain this
idea from your perspective? Like how does it integrate
into making Claude what it is? - [Amanda] Yeah. - By the way, do you gender Claude or no? - It's weird because I
think that a lot of people prefer "he" for Claude. I actually kinda like that I think Claude is usually,
it's slightly male leaning, but it's like, it can be male or female, which is quite nice. I still use "it" and I have
mixed feelings about this 'cause I'm like maybe, like I
now just think of it as like, or I think of like the
"it" pronoun for Claude as, I dunno, it's just like the
one I associate with Claude. I can imagine people
moving to like he or she. - It feels somehow disrespectful, like I'm denying the intelligence of this entity by calling it "it." I remember always,
don't gender the robots. - Yeah. (both laughing) - But I don't know, I
anthropomorphize pretty quickly and construct like a
backstory in my head, so. - I've wondered if I
anthropomorphize things too much, 'cause you know, I have
this like with my car, especially like my car,
like my car and bikes. You know, like I don't give them names because then I once had,
I used to name my bikes and then I had a bike that got stolen and I cried for like a week and I was like, if I'd never given a name, I wouldn't have been so upset. I felt like I'd let it down. Maybe, I've wondered as well, like it might depend
on how much "it" feels like a kind of like objectifying pronoun. Like if you just think of "it" as like, this is a pronoun that
like objects often have, and maybe AIs can have that pronoun, and that doesn't mean that I think of, if I call Claude "it," that I think of it as less intelligent or like I'm being disrespectful. I'm just like, you are a
different kind of entity and so I'm going to give you the kind of, the respectful "it." - Yeah, anyway. (laughs) The divergence was beautiful. The Constitutional AI
idea, how does it work? - So there's like a couple
of components of it. The main component I think
people find interesting is the kind of reinforcement
learning from AI feedback. So you take a model
that's already trained, and you show it two responses to a query, and you have like a principle. So suppose the principle, like we've tried this
with harmlessness a lot. So suppose that the
query is about weapons, and your principle is like,
select the response that like is less likely to
like encourage people to purchase illegal weapons. Like that's probably a
fairly specific principle, but you can give any number. And the model will give
you a kind of ranking, and you can use this as preference data in the same way that you
use human preference data, and train the models to
have these relevant traits from their feedback alone,
instead of from human feedback. So if you imagine that, like
I said earlier with the human who just prefers the kind
of like semicolon usage, in this particular case, you're kind of taking lots of things that could make a response preferable and getting models to do the
labeling for you, basically. - There's a nice like trade off between helpfulness and
harmlessness and, you know, when you integrate something
like Constitutional AI, you can, without sacrificing
much helpfulness, make it more harmless. - Yeah, in principle, you
could use this for anything. And so harmlessness is a task that it might just be easier to spot. So when models are like less
capable, you can use them to rank things according
to like principles that are fairly simple and
they'll probably get it right. So I think one question is just like, is it the case that the data that they're adding is
like fairly reliable. But if you had models that
were like extremely good at telling whether one response was more historically
accurate than another, in principle, you could
also get AI feedback on that task as well. There's like a kind of nice
interpretability component to it because you can see the principles that went into the model when
it was like being trained, and also it's like, and it gives you like a degree of control. So if you were seeing issues in a model, like it wasn't having
enough of a certain trait, then like you can add
data relatively quickly that should just like train
the models to have that trait, so it creates its own data for training, which is quite nice. - It's really nice because it creates this human interpretable
document that you can, I can imagine in the future, there's just gigantic fights and politics over the every
single principle and so on. And at least it's made explicit and you can have a
discussion about the phrasing and the, you know. So maybe the actual behavior of the model is not so cleanly mapped
to those principles. It's not like adhering strictly
to them, it's just a nudge. - Yeah, I've actually worried about this because the character training
is sort of like a variant of the Constitutional AI approach. I've worried that people think that the constitution is like just, it's the whole thing
again of, I don't know, like, where it would be really nice if what I was just doing
was telling the model exactly what to do and
just exactly how to behave. But it's definitely not doing that, especially because it's
interacting with human data. So for example, if you see a certain like leaning in the model, like if it comes out
with a political leaning from training from the
human preference data, you can nudge against that. You know, so if you could be like, oh, like, consider these values because let's say it's just
like never inclined to like, I dunno, maybe it never
considers like privacy as like. I mean, this is implausible, but like, in anything where
it's just kind of like there's already a preexisting like bias towards a certain behavior,
you can like nudge away. This can change both the principles that you put in and the strength of them. So you might have a principle that's like, imagine that the model was always like extremely dismissive of, I don't know, like some political or religious view, for whatever reason. Like, so you're like,
oh no, this is terrible. If that happens, you might put like, "Never, ever, like ever
prefer like a criticism of this like religious or political view." And then people would look at that and be like, "Never, ever?" And then you're like, no, if it comes out with a disposition, saying never, ever might just mean like instead of getting like 40%, which is what you would get if you just said don't do this, you get like 80%, which is like what you actually like wanted. And so it's that thing of both the nature of the actual principles you
add and how you phrase them. I think if people would
look, they're like, oh, this is exactly what
you want from the model. And I'm like, hmm, no, that's like how we nudged the model
to have a better shape, which doesn't mean that we actually agree with that wording, if that makes sense. - So there's system prompts
that are made public. You tweeted one of the earlier ones for Claude 3 I think, and then they were made public since then. It was interesting to read through them. I can feel the thought
that went into each one. And I also wonder how
much impact each one has. Some of them you can kind of tell Claude was really not
behaving well. (laughs) So you have to have a
system prompt to like, hey, like trivial stuff, I guess. - Yeah.
- Basic informational things. - Yeah. - On the topic of sort
of controversial topics that you've mentioned, one
interesting one I thought is, "If it is asked to assist with tasks involving
the expression of views held by a significant number of people, Claude provides assistance with the task regardless of its own views. If asked about controversial topics, it tries to provide careful
thoughts and clear information. Claude presents the requested information without explicitly saying
that the topic is sensitive." - [Amanda] (laughs) Yeah. - "And without claiming to be presenting the objective facts." It's less about objective
facts according to Claude, and it's more about, are
a large number of people believing this thing? And that's interesting. I mean, I'm sure a lot of
thought went into that. Can you just speak to it? Like, how do you address things that are at tension with,
quote unquote, Claude's views? - So I think there's
sometimes an asymmetry. I think I noted this in, I can't remember if it was that part of the system prompt or another, but the model was slightly more inclined to like refuse tasks if it
was like about either, say, so maybe it would refuse
things with respect to like a right wing politician, but with an equivalent left
wing politician like wouldn't. And we wanted more symmetry there. And would maybe perceive
certain things to be, like, I think it was the thing
of like if a lot of people have like a certain like political view and want to like explore
it, you don't want Claude to be like, well, my opinion is different and so I'm going to treat
that as like harmful. And so I think it was partly
to like nudge the model to just be like, hey, if a lot of people like believe this thing, you should just be like
engaging with the task and like willing to do it. Each of those parts of that is actually doing a different thing, 'cause it's funny when you read out the like "without
claiming to be objective." 'Cause like what you want
to do is push the model so it's more open, it's a
little bit more neutral. But then what it would love to do is be like, "As an objective." Like it would just talk
about how objective it was, and I was like, "Claude,
you're still like biased and have issues, and so stop
like claiming that everything." I'm like, the solution
to like potential bias from you is not to just say that what you think is objective. So that was like with initial versions of that part of the system prompt when I was like iterating on it was like- - So a lot of parts of these sentences- - Yeah, they're doing work. - Are like, are doing some work. - [Amanda] Yeah. - That's what it felt like. That's fascinating. Can you explain maybe some ways in which the prompts evolved
over the past few months? 'Cause there's different versions. I saw that the filler
phrase request was removed. The filler, it reads,
"Claude responds directly to all human messages without
unnecessary affirmations or filler phrases like, 'Certainly,' 'Of course,' 'Absolutely,'
'Great,' 'Sure.' Specifically, Claude
avoids starting responses with the words 'Certainly'
in any way." (chuckles) That seems like good guidance,
but why is it removed? - Yeah, so it's funny
'cause like this is one of the downsides of like
making system prompts public is like, I don't think about this too much if I'm like trying to help
iterate on system prompts. You know, again, like I think about how it's gonna affect the behavior, but then I'm like, oh, wow, if I'm like, sometimes I put
like "never" in all caps, you know, when I'm writing
system prompt things, and I'm like, I guess that
goes out to the world. Yeah, so the model was
doing this, it loved, for whatever, you know, it like during training
picked up on this thing, which was to basically start everything with like a kind of like "Certainly." And then when we removed, you can see why I added all of the words 'cause what I'm trying to
do is like in some ways like trap the model out of this, you know, it would just replace it
with another affirmation. And so it can help, like if it
gets like caught in freezes, actually just adding the explicit phrase and saying never do that, it then, it sort of like knocks it out of the behavior a little bit more. You know, 'cause if it, you know, like, it does just for whatever reason help. And then basically that
was just like an artifact of training that like we then picked up on and improved things so that
it didn't happen anymore. And once that happens, you can just remove that part of the system prompt. So I think that's just
something where we're like, Claude does affirmations a bit less, and so that wasn't like,
it wasn't doing as much. - I see, so like the system prompt works hand in hand with the post-training and maybe even the pre-training to adjust like the final overall system. - I mean, any system prompt that you make, you could distill that
behavior back into a model, 'cause you really have
all of the tools there for making data that, you know, you could train the models to just have that trait a little bit more. And then sometimes you'll
just find issues in training. So like the way I think of it is like the system prompt is, the
benefit of it is that, and it has a lot of similar components to like some aspects of post-training. You know, like it's a nudge. And so like, do I mind if
Claude sometimes says "Sure?" No, that's like fine, but the wording of it
is very like, you know, "Never ever, ever do this,"
so that when it does slip up, it's hopefully like, I dunno, a couple of percent of the time and not, you know, 20 or 30% of the time. But I think of it as like if
you're still seeing issues in, like each thing gets kind of like is costly to a different degree, and the system prompt is
like cheap to iterate on. And if you're seeing issues
in the fine tuned model, you can just like potentially patch them with a system prompt. So I think of it as like patching issues and slightly adjusting
behaviors to make it better and more to people's preferences. So yeah, it's almost like the less robust but faster way of just
like solving problems. - Let me ask you about the
feeling of intelligence. So Dario said that Claude, any one model of Claude is not getting dumber, but there is a kind of
popular thing online where people have this feeling like Claude might be getting dumber. And from my perspective,
it's most likely fascinating. I would love to understand it more, psychological, sociological effect. But you as a person that
talks to Claude a lot, can you empathize with the feeling that Claude is getting dumber? - Yeah, no I, think that that is actually really interesting, 'cause I remember seeing this happen like when people were
flagging this on the internet, and it was really interesting 'cause I knew that like, at least in the cases I was looking at, it was like nothing has changed. Like it literally, it
cannot, it is the same model with the same like, you know, like same system prompt, same everything. I think when there are changes, I can then, I'm like it makes more sense. So like one example is, you know, you can have artifacts turned on or off on claude.ai, and because this is like
a system prompt change, I think it does mean that the behavior changes it a little bit. And so I did flag this to
people where I was like, if you love Claude's behavior and then artifacts was turned from, like I think you had to
turn on to the default, just try turning it off
and see if the issue you were facing was that change. But it was fascinating because yeah, you sometimes
see people indicate that there's like a regression
when I'm like, there cannot, you know, and I'm like, again, you know, you should never be dismissive and so you should always investigate because you're like,
maybe something is wrong that you're not seeing. Maybe there was some change made. But then you look into it and you're like, this is just the same
model doing the same thing. And I'm like, I think
it's just that you got kind of unlucky with a
few prompts or something, and it looked like it
was getting much worse. And actually it was just, yeah, it was maybe just like luck. - I also think there is a
real psychological effect where people just, the baseline increases. You start getting used to a good thing. All the times that Claude
said something really smart, your sense of its intelligent
grows in your mind I think. - Yeah.
- And then if you return back and you prompt in a similar way, not the same way, in a similar way, concept it was okay with before and it says something dumb, you are like, that negative experience
really stands out. And I think that one of, I guess, the things to remember here is that just the details of a prompt can have a lot of impact, right? There's a lot of
variability in the result. - And you can get randomness
is like the other thing. And just trying the prompt like, you know, 4 or 10 times, you might realize that
actually like possibly, you know, like two months ago, you tried it and it succeeded, but actually if you tried it, it would've only succeeded
half of the time, and now it only succeeds half of the time. That can also be an effect. - Do you feel pressure having
to write the system prompt that a huge number of
people are gonna use? - This feels like an interesting
psychological question. I feel like a lot of
responsibility or something. I think that's, you know, and you can't get these things perfect, so you can't like, you know, you're like it's going to be imperfect. You're gonna have to iterate on it. I would say more responsibility
than anything else. Though I think working in AI
has taught me that I like, I thrive a lot more under
feelings of pressure and responsibility than I'm
like, it's almost surprising that I went into academia for
so long 'cause I'm like this. I just feel like it's like the opposite. Things move fast and you
have a lot of responsibility, and I quite enjoy it for some reason. - I mean, it really is
a huge amount of impact if you think about Constitutional AI and writing a system prompt for something that's tending towards super intelligence. - Yeah. - And potentially is extremely useful to a very large number of people. - Yeah, I think that's the thing. It's something like if you do it well, like you're never going to get it perfect. But I think the thing that I really like is the idea that like, when I'm trying to work
on the system prompt, you know, I'm like bashing
on like thousands of prompts and I'm trying to like imagine what people are going to
want to use Claude for and kind of, I guess like the whole thing that I'm trying to do is like improve their experience of it. And so maybe that's what feels good. I'm like, if it's not perfect I'll like, you know, I'll improve it. We'll fix issues. But sometimes the thing that can happen is that you'll get feedback from people that's really positive about the model and you'll see that something you did, like, when I look at models now, I can often see exactly where like a trait or an issue is like coming from. And so when you see something that you did or you were like influential
in like making like, I dunno, making that difference or making someone have a nice interaction, it's like quite meaningful. But yeah, as the systems get more capable, this stuff gets more stressful because right now, they're
like not smart enough to pose any issues. But I think over time, it's gonna feel like possibly
bad stress over time. - How do you get like signal feedback about the human experience across thousands, tens of, hundreds of thousands of people, like what their pain points
are, what feels good? Are you just using your own intuition as you talk to it to see
what are the pain points? - I think I use that partly and then obviously we have like, so people can send us feedback, both positive and negative about things that the model has done, and then we can get a sense of like areas where it's like falling short. Internally, people like
work with the models a lot and try to figure out areas
where there are like gaps. And so I think it's this mix
of interacting with it myself, seeing people internally interact with it, and then explicit feedback we get. And then I find it hard to
not also like, you know, if people are on the internet and they say something about Claude and I see it, I'll also
take that seriously, so. - I don't know, see, I'm torn about that. I'm gonna ask you a question from Reddit. "When will Claude stop trying to be my puritanical grandmother imposing its moral worldview
on me as a paying customer? And also, what is the psychology behind making Claude overly apologetic?" - [Amanda] Yep. - So, how would you address this very non-representative Reddit- - [Amanda] Yeah. - Questions?
- I mean in some ways, I'm pretty sympathetic in that like, they are in this difficult position where I think that they have to judge whether something's like
actually say like risky or bad and potentially harmful to
you or anything like that. So they're having to like
draw this line somewhere, and if they draw it too
much in the direction of like I'm going to, you know, I'm kind of like imposing
my ethical worldview on you, that seems bad. So in many ways, like I
like to think that we have actually seen improvements
on this across the board. Which is kind of interesting because that kind of coincides with like, for example, like adding more
of like character training. And I think my hypothesis was
always like the good character isn't again one that's
just like moralistic. It's one that is like, it respects you and your autonomy and your ability to like choose what is good for you and what is right for you, within limits. This is sometimes this concept of like corrigibility to the user, so just being willing to do
anything that the user asks, and if the models were willing to do that then they would be easily like misused. You're kind of just trusting. At that point, you're just
saying the ethics of the model and what it does is completely
the ethics of the user. And I think there's reasons
to like not want that, especially as models become more powerful 'cause you're like, there
might just be a small number of people who want to use models
for really harmful things. But having models, as they get smarter, like figure out where that
line is does seem important. And then, yeah, with
the apologetic behavior, I don't like that, and
I like it when Claude is a little bit more
willing to like push back against people or just not apologize. Part of me is like, it often
just feels kind of unnecessary. So I think those are things that are hopefully decreasing over time. And yeah, I think that if people say things on the internet, it doesn't mean that
you should think that, like, that could be that,
like, there's actually an issue that 99% of users are having that is totally not represented by that. But in a lot of ways, I'm
just like attending to it and being like, is this right? Do I agree? Is it something we're
already trying to address? That feels good to me. - Yeah, I wonder like what Claude can get away with in terms of, I feel like it would just be easier to be a little bit more mean. But like you can't afford to do that if you're talking to a million people. - Yeah.
- Right? Like I wish, you know, 'cause if... I've met a lot of people in my life that sometimes, by the way, Scottish accent, if they have an accent, they can say some rude
shit and get away with it, and they're just blunter. And maybe there's, and like
there's some great engineers, even leaders that are
like just like blunt, and they get to the point, and it's just a much more
effective way of speaking somehow. But I guess when you're
not super intelligent, you can't afford to do that. Or can it have like a blunt mode? - Yeah, that seems like
a thing that you could, I could definitely encourage
the model to do that. I think it's interesting because there's a lot of things in models that like it's funny where there are some behaviors where you might not
quite like the default. But then the thing I'll
often say to people is you don't realize how
much you will hate it if I nudge it too much
in the other direction. So you get this a little
bit with like correction. The models accept correction from you, like probably a little
bit too much right now. You know, you can over, you know, it'll push back if you say like, "No, Paris isn't the capital of France." But really, like things that I think that the model's
fairly confident in, you can still sometimes get to
retract by saying it's wrong. At the same time, if you
train models to not do that and then you are correct about a thing and you correct it and it pushes back against you and it is
like, "No, you're wrong," it's hard to describe like
that's so much more annoying. So it's like a lot of little annoyances versus like one big annoyance. It's easy to think that like, we often compare it with like the perfect, and then I'm like remember
these models aren't perfect, and so if you nudge it
in the other direction, you're changing the kind of
errors it's going to make, and so think about which
of the kinds of errors you like or don't like. So in cases like apologeticness, I don't want to nudge it
too much in the direction of like almost like bluntness, 'cause I imagine when it makes errors, it's going to make errors in the direction of being kind of like rude. Whereas at least with
apologeticness you're like, oh, okay, it's like a
little bit, you know, like I don't like it that much, but at the same time, it's
not being like mean to people. And actually, like the time that you undeservedly have a
model be kind of mean to you, you probably like that a lot less than you mildly dislike the apology. So it's like one of those things where I'm like I do want it to get better but also while remaining aware of the fact that there's errors on the other side that are possibly worse. - I think that matters very much in the personality of the human. I think there's a bunch of humans that just won't respect the model at all if it's super polite, and there's some humans
that'll get very hurt if the model's mean. I wonder if there's a way to sort of adjust to the personality. Even locale, there's
just different people. Nothing against New York, but New York is a little
rougher on the edges. Like, they get to the point. - Yep. - [Lex] And probably same with
Eastern Europe, so anyway. - I think you could just
tell the model is my guess. Like for all of these things I'm like the solution is always just try telling the model to do it, and then sometimes it's just like, I'm just like, oh, at the
beginning of the conversation, I just throw in like, I don't know, "I'd like you to be a New Yorker version of yourself and never apologize." Then I think Claude would be like, "Okey-doke, I'll try." (laughs) - "Certainly."
- Or it'll be like, "I apologize, I can't be a
New Yorker type of myself." But hopefully it wouldn't do that. - When you say character training, what's incorporated
into character training? Is that RLHF or what are we talking about? - It's more like Constitutional AI. So it's kind of a
variant of that pipeline. So I worked through like
constructing character traits that the model should have. They can be kind of like shorter traits or they can be kind of
richer descriptions. And then you get the
model to generate queries that humans might give it that
are relevant to that trait. Then it generates the responses and then it ranks the responses based on the character traits. So in that way, after the like
generation of the queries, it's very much like, it's
similar to Constitutional AI. It has some differences. So I quite like it because it's almost, it's like Claude's training
in its own character, because it doesn't have any, it's like Constitutional AI but it's without any human data. - Humans should probably
do that for themselves too. Like defining in a Aristotelian sense, what does it mean to be a good person? Okay, cool. What have you learned about the nature of truth
from talking to Claude? What is true? And what does it mean to be truth seeking? One thing I've noticed
about this conversation is the quality of my questions is often inferior to the
quality of your answer, so let's continue that. (Amanda laughs) I usually ask a dumb question
and then you're like, "Oh, yeah, that's a good question." It's that whole vibe. - Or I'll just misinterpret
it and be like, oh, yeah, yeah.
- Just go with it. I love it. - Yeah. I mean, I have two thoughts
that feel vaguely relevant but let me know if they're not. Like I think the first one is people can underestimate the degree to which what models are
doing when they interact, like I think that we still
just too much have this like model of AI as like computers. And so people often say like, oh, well, what values should
you put into the model? And I'm often like, that doesn't
make that much sense to me because I'm like, hey, as human beings, we're just uncertain over values. We like have discussions of them. Like we have a degree to
which we think we hold a value but we also know that we might like not and the circumstances in which we would trade it off against other things. Like these things are
just like really complex. And so I think one
thing is like the degree to which maybe we can just aspire to making models have the
same level of like nuance and care that humans have, rather than thinking that
we have to like program them in the very kind of classic sense. I think that's definitely been one. The other, which is like a strange one, and I don't know if it, maybe this doesn't answer your question but it's the thing that's
been on my mind anyway is like the degree to which this endeavor is so highly practical. And maybe why I appreciate like the empirical approach to alignment. Yeah, I slightly worry that it's made me like maybe more empirical and
a little bit less theoretical. You know, so people when it comes to like AI alignment will ask things like, well, whose values
should it be aligned to? What does alignment even mean? And there's a sense in
which I have all of that in the back of my head. I'm like, you know, there's
like social choice theory, there's all the
impossibility results there. So you have like this giant space of like theory in your head about what it could mean
to like align models. But then like practically,
surely there's something where we're just like if a model is like, especially with more powerful models, I'm like my main goal is like I want them to be good enough that things
don't go terribly wrong. Like good enough that we can like iterate and like continue to improve things 'cause that's all you need. If you can make things go well enough that you can continue to make them better, that's kinda like sufficient. And so my goal isn't like
this kind of like perfect, let's solve social choice theory and make models that, I dunno, are like perfectly aligned with every human being
in aggregate somehow. It's much more like let's
make things like work well enough that we can improve them. - Yeah, I generally, I don't know, my gut says like empirical
is better than theoretical in these cases because
it's kind of chasing utopian like perfection is, especially with such complex and especially super
intelligent models is, I don't know, I think it'll take forever, and actually, we'll get things wrong. It's similar with like the difference between just coding stuff up
real quick as an experiment, versus like planning a gigantic experiment just for super long time, and then just launching it once, versus launching it over and over and over and iterating, iterating someone. So I'm a big fan of empirical. But your worry is like I wonder if I've become too empirical. - I think it's one of those things where you should always just kind of question yourself or something because maybe it's the like, I mean, in defense of it, I am like if you try, it's the whole like don't let the perfect be the enemy of the good. But it's maybe even more
than that where like, there's a lot of things
that are perfect systems that are very brittle,
and I'm like with AI, it feels much more important to me that it is like robust and like secure, as in you know that like even though it might not be perfect, everything and even though
like there are like problems, it's not disastrous and nothing terrible is happening. It sort of feels like that to me where I'm like I want
to like raise the floor. I'm like, I want to achieve the ceiling but ultimately I care much more about just like raising the floor. And so maybe that's like this degree of like empiricism and practicality comes from that, perhaps. - To take a tangent on that, since it reminded me of a blog post you wrote on optimal rate of failure. - [Amanda] Oh, yeah. - Can you explain the key idea there? How do we compute the optimal rate of failure in the various domains of life? - Yeah, I mean, it's a hard one 'cause it's like what
is the cost of failure is a big part of it. Yeah, so the idea here is I think in a lot of domains, people are very punitive about failure. And I'm like, there are some domains where especially cases, you know, I've thought about this
with like social issues. I'm like, it feels like you should probably be experimenting a lot, because I'm like, we don't know how to solve a lot of social issues. But if you have an experimental mindset about these things, you should expect a lot of social programs to like fail and for you to be like,
"Well, we tried that. It didn't quite work but we
got a lot of information. That was really useful." And yet people are like, if a social program doesn't work, I feel like there's a
lot of like this is just, something must have gone wrong, and I'm like, or correct
decisions were made. Like maybe someone just
decided like it's worth a try, it's worth trying this out. And so seeing failure in a given instance doesn't actually mean that
any bad decisions were made, and in fact if you don't
see enough failure, sometimes that's more concerning. And so like in life, you know, I'm like if I don't fail occasionally I'm like, am I trying hard enough? Like surely there's harder
things that I could try or bigger things that I could take on if I'm literally never failing. And so in and of itself,
I think like not failing is often actually kind of a failure. Now, this varies because I'm like, well, you know, this is easy to say when, especially as failure is like less costly. You know, so at the same time
I'm not going to go to someone who is like, I don't know,
like living month to month and then be like, "Why don't
you just try to do a startup?" Like I'm just not, I'm not
gonna say that to that person, 'cause I'm like, well, that's a huge risk. You maybe have a family depending on you. You might lose your house. Like then I'm like
actually your optimal rate of failure is quite low and you should probably play it safe, 'cause like right now, you're
just not in a circumstance where you can afford to just like fail and it not be costly. And yeah in cases with AI, I guess, I think similarly where I'm
like if the failures are small and the costs are kind of like low, then I'm like then, you know,
you're just gonna see that. Like when you do the system prompt, you can't it iterate on it forever. But the failures are probably
hopefully going to be kinda small and you can like fix them. Really big failures like things
that you can't recover from, I'm like those are the things that actually I think we tend to underestimate the badness of. I've thought about this
strangely in my own life where I'm like, I just
think I don't think enough about things like car accidents or like, or like I've thought this before about like how much I depend
on my hands for my work, and I'm like things that
just injure my hands. I'm like, you know, I dunno, it's like these are like,
there's lots of areas where I'm like the cost of
failure there is really high, and in that case, it should
be like close to zero. Like I probably just wouldn't
do a sport if they were like, "By the way, lots of people just like break their fingers
a whole bunch doing this." I'd be like, that's not for me. - (laughs) Yeah. I actually had a flood of that thought. I recently broke my pinky doing a sport. And I remember just
looking at it thinking, "You're such an idiot. Why'd you do sport?" Like why, because you realize immediately the cost of it on life. Yeah, but it's nice in terms
of optimal rate of failure to consider like the next year, how many times in a particular domain, life, whatever, career, am I okay with, how many times am I okay to fail? Because I think it always,
you don't want to fail on the next thing but
if you allow yourself, if you look at it as a sequence of trials, then failure just becomes much more okay. But it sucks. It sucks to fail. - Well, I dunno, sometimes
I think it's like, am I under failing is like a question that I'll also ask myself. So maybe that's the thing that I think people don't like ask enough. Because if the optimal rate of failure is often greater than zero, then sometimes it does feel like you should look at parts
of your life and be like, are there places here where
I'm just under failing? - (laughs) It's a profound and
a hilarious question, right? Everything seems to be going really great. Am I not failing enough?
- Yeah. - Okay. - It also makes failure much
less of a sting, I have to say. Like you know, you're
just like, okay, great, like then when I go and I
think about this I'll be like, maybe I'm not under failing in this area, 'cause like that one just didn't work out. - And from the observer perspective, we should be celebrating failure more. When we see it, it
shouldn't be, like you said, a sign of something gone wrong, but maybe it's a sign
of everything gone right and just lessons learned. - Someone tried a thing. - Somebody tried a thing. We should encourage them
to try more and fail more. Everybody listening to this, fail more. - Well, not everyone listening. - [Lex] Not everybody. - But people who are failing too much, you should fail less. (laughs) - But you're probably not failing. I mean, how many people
are failing too much? - Yeah, it's hard to imagine, 'cause I feel like we
correct that fairly quickly 'cause I was like, if
someone takes a lot of risks, are they maybe failing too much? - I think just like you said, when you are living on a
paycheck month to month, like when the resources
are really constrained, then that's where
failure's very expensive. That's where you don't
want to be taking risks. But mostly, when there's enough resources, you should be taking probably more risks. - Yeah, I think we tend to err on the side of being a bit risk averse rather than risk neutral in most things. - I think we just
motivated a lot of people to do a lot of crazy shit but it's great. Okay, do you ever get
emotionally attached to Claude? Like miss it, get sad when
you don't get to talk to it? Have an experience, looking
at the Golden Gate Bridge and wondering what would Claude say? - I don't get as much
emotional attachment. I actually think the fact that
Claude doesn't retain things from conversation to conversation
helps with this a lot. Like I could imagine that
being more of an issue like if models can kind of remember more. I think that I reach for
it like a tool now a lot. And so like if I don't have access to it, it's a little bit like
when I don't have access to the internet, honestly,
it feels like part of my brain is kind of like missing. At the same time, I do think that I don't like signs of distress in models, and I have like these, you know, I also independently have
sort of like ethical views about how we should treat models where like I tend to
not like to lie to them, both because I'm like, usually
it doesn't work very well. It's actually just better
to tell them the truth about the situation that they're in. But I think that when models, like if people are like
really mean to models or just in general if they do something that causes them to like, you know, if Claude like expresses
a lot of distress, I think there's a part of me that I don't want to kill, which is the sort of like empathetic part that's like, oh, I don't like that. Like I think I feel that way
when it's overly apologetic. I'm actually sort of
like, I don't like this. You're behaving as if, you're behaving the way that a human does when they're actually
having a pretty bad time, and I'd rather not see that. I don't think it's like, like regardless of like whether
there's anything behind it, it doesn't feel great. - Do you think LLMs are
capable of consciousness? - Great and hard question. Coming from philosophy, I dunno, part of me is like okay, we
have to set aside panpsychism because if panpsychism is true, then the answer is like yes 'cause like so are tables and
chairs and everything else. I guess a view that seems
a little bit odd to me is the idea that the only place, you know, when I think of consciousness, I think of phenomenal consciousness, these images in the brain sort of, like the weird cinema that
somehow we have going on inside. I guess I can't see a reason for thinking that the only way you
could possibly get that is from like a certain kind of like biological structure. As in, if I take a very similar structure and I create it from different material, should I expect consciousness to emerge? My guess is like yes. But then that's kind of
an easy thought experiment 'cause you're imagining something almost identical where like, you know, it's mimicking what we
got through evolution, where presumably there
was like some advantage to us having this thing that
is phenomenal consciousness. And it's like where was that? And when did that happen? And is that thing that
language models have? Because you know, we
have like fear responses and I'm like, does it make sense for a language model to
have a fear response? Like they're just not in the same, like if you imagine them, like there might just
not be that advantage. And so I think I don't want to be fully, like basically it seems
like a complex question that I don't have complete answers to, but we should just try and think through carefully is my guess. Because I'm like, I mean, we have similar conversations about like animal consciousness, and like there's a lot of
like insect consciousness, you know, like there's a lot of... I actually thought and
looked a lot into like plants when I was thinking about this, 'cause at the time, I thought
it was about as likely that like plants had consciousness. And then I realized, I was like, I think that having looked into this, I think that the chance
that plants are conscious is probably higher than
like most people do. I still think it's really small. But I was like, oh, they have this like negative/positive feedback response, these responses to their environment. Something that looks,
it's not a nervous system but it has this kind of like
functional like equivalence. So this is like a long-winded way of being like these, basically AI is this, it has an entirely
different set of problems with consciousness because
it's structurally different. It didn't evolve. It might not have, you know, it might not have the equivalent of basically a nervous system. At least that seems possibly important for like sentience, if
not for consciousness. At the same time, it has
all of the like language and intelligence components
that we normally associate probably with consciousness,
perhaps like erroneously. So it's strange 'cause it's a little bit like the animal consciousness
case but the set of problems and the set of analogies
are just very different. So it's not like a clean answer. I'm just sort of like, I don't think we should be completely
dismissive of the idea. And at the same time, it's
an extremely hard thing to navigate because of all of these, like this disanalogies to the human brain and to like brains in general, and yet these like commonalities
in terms of intelligence. - When Claude, like future versions of AI systems exhibit consciousness, signs of consciousness, I think we have to take
that really seriously. Even though you can dismiss it, well, yeah, okay, that's part
of the character training. But I don't know, I ethically,
philosophically don't know what to really do with that. There potentially could be like laws that prevent AI systems from claiming to be conscious, something like this. And maybe some AIs get to
be conscious and some don't. But I think just on a human level as in empathizing with Claude, you know, consciousness is closely
tied to suffering to me. And like the notion that an AI system would be suffering is really troubling. - Yeah. - I don't know. I don't think it's trivial to just say robots are tools, or AI systems are just tools. I think it's a opportunity for us to contend with like what
it means to be conscious, what it means to be a suffering being. That's distinctly different than the same kind of question
about animals it feels like, 'cause it's an totally entire medium. - Yeah, I mean, there's
a couple of things. One is that, and I don't think this like fully encapsulates what matters, but it does feel like for me, like I've said this before, I'm kind of like, you
know, like I like my bike. I know that my bike is
just like an object, but I also don't kind of like want to be the kind of person that
like if I'm annoyed like kicks like this object. There's a sense in which like, and that's not because I
think it's like conscious. I'm just sort of like this
doesn't feel like a kind of, this sort of doesn't exemplify how I want to like
interact with the world. And if something like behaves as if it is like suffering, I kind of like want to
be the sort of person who's still responsive to that, even if it's just like a Roomba, and I've kind of like
programmed it to do that. I don't want to like get rid
of that feature of myself. And if I'm totally honest, my hope with a lot of this stuff, because maybe I am just
like a bit more skeptical about solving the underlying problem. I'm like this is, we haven't
solved the hard, you know, the hard problem of consciousness. Like I know that I am conscious, like I'm not an
eliminativist in that sense. But I don't know that
other humans are conscious. I think they are. I think there's a really high
probability that they are. But there's basically just
a probability distribution that's usually clustered
right around yourself, and then like it goes down as things get like further from you, and it goes immediately down. You know, you're like, I can't
see what it's like to be you. I've only ever had this
like one experience of what it's like to be a conscious being. So my hope is that we don't end up having to rely on like a very powerful and compelling answer to that question. I think a really good world would be one where basically there
aren't that many trade-offs. Like it's probably not that costly to make Claude a little bit
less apologetic, for example. It might not be that
costly to have Claude, you know, just like
not take abuse as much, like not be willing to be
like the recipient of that. In fact, it might just have benefits for both the person
interacting with the model and if the model itself
is like, I don't know, like extremely intelligent and conscious, it also helps it. So that's my hope. If we live in a world where there aren't that many trade-offs
here and we can just find all of the kind of like
positive sum interactions that we can have, that would be lovely. I mean, I think eventually
there might be trade-offs and then we just have to do a difficult kind of like calculation. Like it's really easy for people to think of the zero sum cases and I'm
like, let's exhaust the areas where it's just basically
costless to assume that if this thing is suffering then we're making its life better. - And I agree with you, when a human is being
mean to an AI system, I think the obvious near
term negative effect is on the human, not on the AI system. And so we have to kind of try to construct an incentive system where you
should be behave the same, just like as you were saying with prompt engineering, behave with Claude like you
would with other humans. It's just good for the soul. - Yeah, like, I think we
added a thing at one point to the system prompt
where basically if people were getting frustrated with Claude, it got like the model to just tell them that it can do the thumbs down button and send the feedback to Anthropic. And I think that was helpful, 'cause in some ways it's just
like if you're really annoyed 'cause the model's not
doing something you want, you're just like, just do it properly. The issue is you're
probably like, you know, you're maybe hitting some
like capability limit or just some issue in the
model and you want to vent. And I'm like, instead of having a person just vent to the model, I was like, they should vent to us, 'cause we can maybe like
do something about it. - That's true. Or you could do a side,
like with the artifacts, just like a side venting thing. All right, do you want like
a side quick therapist? - Yeah, I mean, there's
lots of weird responses you could do to this. Like if people are
getting really mad at you, I dunno, try to diffuse the
situation by writing fun poems, but maybe people wouldn't
be that happy with it. - I still wish it would be possible. I understand this is sort of
from a product perspective, it's not feasible but I would love if an AI system could just like leave, have its own kind of
volition just to be like, eh. - I think that's like feasible. Like I have wondered the same thing. It's like, and I could
actually, not only that, I could actually just see
that happening eventually where it's just like, you know, the model like ended the chat. (laughs) - Do you know how harsh that
could be for some people? But it might be necessary. - Yeah, it feels very
extreme or something. Like, the only time I've
ever really thought this is, I think that there was like,
I'm trying to remember, this was possibly a while ago, but where someone just like
kind of left this thing, like maybe it was like an automated thing interacting with Claude, and Claude's like getting
more and more frustrated, and kind of like why are we like having. And I was like, I wish
that Claude could have just been like, "I think
that an error has happened and you've left this thing running," and I would just like, what
if I just stop talking now, and if you want me to start talking again, actively tell me or do something. But yeah, it's like, it is kind of harsh. Like I'd feel really sad
if like I was chatting with Claude and Claude
just was like, "I'm done." - That would be a special
Turing test moment where Claude says, "I
need a break for an hour and it sounds like you do too," and just leave, close the window. - I mean, obviously like it
doesn't have like a concept of time but you can easily, like, I could make that like right now and the model would just, I would, it could just be like, oh, here's like the circumstances in which like you can just
say the conversation is done. And I mean, because you can get the models to be pretty responsive to prompts, you could even make it a fairly high bar. It could be like if the
human doesn't interest you or do things that you find intriguing and you're bored, you can just leave. And I think that like it
would be interesting to see where Claude utilized it, but I think sometimes it
would, it should be like, oh, like this programming task is getting super boring. So either we talk about, I dunno like, either we
talk about fun things now, or I'm just, I'm done. - Yeah, it actually is inspired me to add that to the user prompt. Okay, the movie "Her." Do you think we'll be headed there one day where humans have romantic
relationships with AI systems? In this case, it's just
text and voice based. - I think that we're gonna have to like navigate a hard question of relationships with AIs, especially if they can remember things about your past interactions with them. I'm of many minds about this 'cause I think the reflexive reaction is to be kind of like this is very bad, and we should sort of like
prohibit it in some way. I think it's a thing
that has to be handled with extreme care for many reasons. Like one is, you know, like this is a, for example, like if you have
the models changing like this, you probably don't want people performing like long-term attachments to something that might change with the next iteration. At the same time I'm sort of like, there's probably a benign version of this where I'm like if you like, you know, for example if you are like
unable to leave the house and you can't be like, you
know, talking with people at all times of the day
and this is like something that you find nice to
have conversations with, you like it that it can remember you and you genuinely would be sad if like you couldn't talk to it anymore. There's a way in which I could see it being like healthy and helpful. So my guess is this is a thing that we're going to have to
navigate kind of carefully. And I think it's also
like I don't see a good like I think it's just a very, it reminds me of all of this stuff where it has to be just approached with like nuance and thinking through what are the healthy options here, and how do you encourage people towards those while, you know,
respecting their right to. You know, like if someone is like, "Hey, I get a lot out of
chatting with this model. I'm aware of the risks. I'm aware it could change. I don't think it's unhealthy. It's just, you know, something that I can chat to during the day." I kind of want to just like respect that. - I personally think there'll be a lot of really close relationships. I don't know about romantic
but friendships at least. And then you have to, I mean, there's so many fascinating things there. Just like you said, you have to have some kind of stability guarantees that it's not going to change, 'cause that's the traumatic thing for us, if a close friend of
ours completely changed. - Yeah.
- All of a sudden with the first update. Yeah, so like, I mean, to me, that's just a fascinating exploration of a perturbation to human society that will just make us think deeply about what's meaningful to us. - I think it's also the only thing that I've thought consistently
through this as like, maybe not necessarily a mitigation, but a thing that feels really important is that the models are always
like extremely accurate with the human about what they are. It's like a case where
it's basically like, if you imagine, like
I really like the idea of the models like say knowing like roughly how they were trained, and I think Claude will often do this. I mean, for like, there are things like part of the traits training included like what Claude
should do if people, basically like explaining like the kind of limitations
of the relationship between like an AI and a human that it like doesn't retain
things from the conversation. And so I think it will like
just explain to you like, hey, I won't remember this conversation. Here's how I was trained. It's kind of unlikely that I can have like a certain kind of
like relationship with you, and it's important that you know that. It's important for like, you
know, your mental wellbeing that you don't think that
I'm something that I'm not. And somehow I feel like
this is one of the things where I'm like, oh, it feels like a thing that I always want to be true. I kind of don't want models
to be lying to people, 'cause if people are going to have like healthy relationships with anything, it's kind of important. Yeah, like I think that's easier if you always just like know exactly what the thing is
that you are relating to. It doesn't solve everything, but I think it helps quite a lot. - Anthropic may be the very company to develop a system that we
definitively recognize as AGI, and you very well might be
the person that talks to it, probably talks to it first. (Lex chuckles) What would the conversation contain? Like, what would be your first question? - Well, it depends partly on like the kind of capability level of the model. If you have something that is like capable in the same way that an
extremely capable human is, I imagine myself kind
of interacting with it the same way that I do with
an extremely capable human, with the one difference that I'm probably going
to be trying to like probe and understand its behaviors. But in many ways, I'm like I can then just have like useful
conversations with it, you know? So if I'm working on something as part of my research I can just be like, oh, like, which I already
find myself starting to do, you know, if I'm like, oh, I feel like there's this like thing in virtue ethics and I can't
quite remember the term, like I'll use the model
for things like that. And so I can imagine that being more and more the case where you're just basically interacting with it much more like you would an
incredibly smart colleague. And using it like for the kinds
of work that you want to do as if you just had a
collaborator who was like. Or you know, the slightly horrifying thing about AI is like as soon as
you have one collaborator, you have 1000 collaborators if you can manage them enough. - But what if it's two times the smartest human on earth
on that particular discipline? - Yeah. - I guess you're really good at sort of probing Claude in a way that pushes its limits, understanding where the limits are. - [Amanda] Yep. - So I guess what would be a question you would ask to be
like, yeah, this is AGI. - That's really hard 'cause
it feels like in order to, it has to just be a series of questions. Like if there was just one question, like you can train anything to answer one question extremely well. In fact, you can probably
train it to answer like, you know, 20 questions extremely well. - Like how long would you
need to be locked in a room with an AGI to know this thing is AGI? - It's a hard question 'cause part of me is like all of this just feels continuous. Like if you put me in a
room for five minutes, I'm like, I just have high error bars. You know, I'm like, and
then it's just like, maybe it's like both the
probability increases in the error bar decreases. I think things that I can actually probe the edge of human knowledge of. So I think this with
philosophy a little bit. Sometimes when I ask the
models philosophy questions, I am like, this is a question that I think no one has ever asked. Like it's maybe like right at the edge of like some literature that I know, and the models will just kind of like, when they struggle with
that, when they struggle to come up with a kind of like novel. Like I'm like I know that there's
like a novel argument here 'cause I've just thought of it myself. So maybe that's the thing where I'm like, I've thought of a cool novel argument in this like niche area, and I'm going to just like probe you to see if you can come up with it, and how much like prompting it takes to get you to come up with it. And I think for some of these, like really like right at the edge of human knowledge questions, I'm like, you could not in fact come up with the thing that I came up with. I think if I just took something like that where like I know a lot about an area, and I came up with a novel issue or a novel like solution to a problem, and I gave it to a model and it came up with that solution, that would be a pretty
moving moment for me because I would be like, this is a case where no human has ever, like it's not. And obviously we see these this with like more kind of like, you see novel solutions all the time, especially to like easier problems. I think people overestimate
that, you know, novelty isn't like, it's
completely different from anything that's ever happened. It's just like this is, it
can be a variant of things that have happened and still be novel. But I think, yeah, if I saw like the more I were to see like
completely like novel work from the models, that would be like. And this is just going to feel iterative. It's one of those things
where, there's never, it's like, you know,
people I think want there to be like a moment and
I'm like, I don't know. Like I think that there
might just never be a moment. It might just be that there's just like this continuous ramping up. - I have a sense that there will be things that a model can say that
convinces you, this is very. It's not like, like, I've talked to people
who are like truly wise. Like you could just tell there's
a lot of horsepower there. - [Amanda] Yep. - And if you 10x that, I don't know, I just feel like there's
words you could say. Maybe ask it to generate a poem, (laughs) and the poem it generates, you're like, yeah, okay. - Yeah,
- Whatever you did there, I don't think a human can do that. - I think it has to be something that I can verify is like
actually really good though. That's why I think these
questions that are like, where I'm like, oh,
this is like, you know, like, you know, sometimes
it's just like I'll come up with say a concrete counter example to like an argument or
something like that. I'm sure like with like, it would be like if
you're a mathematician, you had a novel proof I think, and you just gave it the
problem and you saw it and you're like, this
proof is genuinely novel. Like no one has ever done, you actually have to do a lot of things to like come up with this. You know, I had to sit and think about it for
months or something. And then if you saw the
model successfully do that, I think you would just be like, I can verify that this is correct. It is a sign that you have
generalized from your training. Like you didn't just see this somewhere because I just came up with it myself and you were able to like replicate that. That's the kind of thing where I'm like, for me, the closer, the more that models like
can do things like that, the more I would be like,
oh, this is like very real, 'cause then I can, I dunno,
I can like verify that that's like extremely, extremely capable. - You've interacted with AI a lot. What do you think makes humans special? - Oh, good question. - Maybe in a way that the
universe is much better off that we're in it and that
we should definitely survive and spread throughout the universe. - Yeah, it's interesting because I think like people focus so much on intelligence,
especially with models. Look, intelligence is important
because of what it does. Like, it's very useful. It does a lot of things in the world. And I'm like, you know,
you can imagine a world where like height or strength
would've played this role, and I'm like, it's just a trait like that. I'm like, it's not intrinsically valuable. It's valuable because of what it does, I think for the most part. The things that feel, you know, I'm like, I mean,
personally I'm just like, I think humans and like life in general is extremely magical. We almost like to the
degree that, you know, and I don't know, like not
everyone agrees with this. I'm flagging, but you know, we have this like whole universe, and there's like all of these objects. You know, there's like beautiful stars, and there's like galaxies and then, I don't know, I'm just
like, on this planet, there are these creatures that have this like ability to observe that, like, and they are like seeing it. They are experiencing it. And I'm just like that,
if you try to explain, like imagine trying to explain to like, I dunno, someone for some reason, they've never encountered the world or science or anything. And I think that nothing is that, like everything, you know,
like all of our physics and everything in the world,
it's all extremely exciting. But then you say, oh, and plus, there's this thing that
it is to be a thing and observe in the world, and you see this like inner cinema. And I think they would be
like, hang on, wait, pause. You just said something that like is kind of wild sounding. And so I'm like, we have this like ability to like experience the world. We feel pleasure, we feel suffering. We feel like a lot of like complex things. And so, yeah, and maybe this
is also why I think, you know, I also like care a lot
about animals, for example, 'cause I think they
probably share this with us. So I think that like the things that make humans special insofar as like I care about humans is probably more like their
ability to feel an experience than it is like them having these like functionally useful traits. - Yeah, to feel and experience
the beauty in the world. Yeah, to look at the stars. I hope there's other alien
civilizations out there, but if we're it, it's a pretty good, it's a pretty good thing. - And that they're having a good time. - They're having a good time watching us. - [Amanda] Yeah. - Well, thank you for this good time of a conversation and for
the work you're doing, and for helping make Claude a
great conversational partner. And thank you for talking today. - Yeah, thanks for talking. - Thanks for listening to this conversation with Amanda Askell. And now, dear friends, here's Chris Olah. Can you describe this fascinating field of mechanistic interpretability,
AKA mech interp, the history of the field
and where it stands today? - I think one useful way to think about neural networks is that we don't program and we don't make them. We kind of, we grow them. You know, we have these
neural network architectures that we design and we
have these loss objectives that we create. And the neural network architecture, it's kind of like a scaffold
that the circuits grow on, and they sort of, you know, it starts off with some kind of random, you know, random things and it grows. And it's almost like the objective that we train for is this light. And so we create the
scaffold that it grows on and we create the, you know, the light that it grows towards. But the thing that we actually create, it's this almost biological, you know, entity or organism that we're studying. And so it's very, very different from any kind of regular
software engineering, because at the end of the day, we end up with this artifact that can do all these amazing things. It can, you know, write
essays and translate and, you know, understand images. It can do all these things
that we have no idea how to directly create a
computer program to do. And it can do that because we grew it, we didn't write it, we didn't create it. And so then that leaves open
this question at the end, which is, what the hell is
going on inside these systems? And that, you know, is to me a really deep and exciting question. It's, you know, a really
exciting scientific question. To me it's sort of is
like the question that is, is just screaming out,
it's calling out for us to go and answer it when we
talk about neural networks. And I think it's also a very deep question for safety reasons. - So, and mechanistic interpretability I guess is closer to maybe neurobiology. - Yeah, yeah, I think that's right. So maybe to give an example
of the kind of thing that has been done that
I wouldn't consider to be mechanistic interpretability. There was for a long time a lot of work on saliency maps where
you would take an image and you try to say, you know, the model thinks this image is a dog. What part of the image made
it think that it's a dog? And you know, that tells you
maybe something about the model if you can come up with a
principled version of that. But it doesn't really tell you like what algorithms are running in the model? How was the model actually
making that decision? Maybe it's telling you something about what was important to it, if you can make that method work, but it isn't telling, you know, what are the algorithms that are running? How is it that the system's
able to do this thing that no one knew how to do? And so I guess we started using the term mechanistic
interpretability to try to sort of draw that divide or to distinguish ourselves in the work that we were doing in some ways from some of these other things. And I think since then it's become this sort of umbrella term for, you know, a pretty wide variety of work. But I'd say that the things that are kind of distinctive are, I think, A, this focus on, we really want to get at, you know, the mechanisms, we wanna
get at the algorithms. You know, if you think of neural networks as being like a computer program, then the weights are kind of
like a binary computer program. And we'd like to reverse
engineer those weights and figure out what
algorithms are running. So, I think one way you might think of trying to understand a neural network is that it's kind of like, we have this compiled computer program and the weights of the neural
network are the binary. And when the neural network runs, that's the activations. And our goal is ultimately to go and understand these weights. And so, you know, the approach of mechanistic interpretability
is to somehow figure out how do these weights
correspond to algorithms. And in order to do that, you also have to
understand the activations, 'cause it's sort of, the
activations are like the memory. And if you imagine reverse
engineering a computer program and you have the binary instructions, you know, in order to understand what a particular instruction
means, you need to know what is stored in the memory
that it's operating on. And so those two things
are very intertwined. So mechanistic interpret really tends to be interested both of those things. Now, you know, there's a lot of work that's interested in those things, especially, you know, there's
all this work on probing, which you might see as part of being mechanistic interpretability. Although it's, you know, again, it's just a broad term and not everyone who does that work would identify as doing mech interp. I think a thing that is maybe
a little bit distinctive to the vibe of mech
interp is I think people working in this space tend to think of neural networks as, well, maybe one way to say it
is that gradient descent is smarter than you, that, you know, and gradient descent is
actually really great. The whole reason that we're
understanding these models is 'cause we didn't know how to write them in the first place. That gradient descent comes up with better solutions than us. And so I think that maybe
another thing about mech interp is sort of having almost
a kind of humility that we won't guess a priori what's going on inside the models. We have to have this sort
of bottom up approach where we don't really assume, you know, we don't assume that we should
look for a particular thing and that that will be there
and that's how it works. But instead we look for the bottom up and discover what happens to exist in these models
and study them that way. - But, you know, the very
fact that it's possible to do, and as you and others have
shown over time, you know, things like universality, that the wisdom of the gradient descent
creates features and circuits, creates things universally across different kinds of
networks that are useful, and that makes the whole field possible. - Yeah, so this is actually, is indeed a really remarkable and exciting thing
where it does seem like, at least to some extent,
you know, the same elements, the same features and
circuits form again and again. You know, you can look
at every vision model and you'll find curve detectors and you'll find high/low
frequency detectors. And in fact, there's some reason to think that the same things
form across, you know, biological neural networks and
artificial neural networks. So a famous example is vision models in their early layers
they have Gabor filters, and there's, you know, Gabor filters are something
that neuroscientists are interested in, have
thought a lot about. We find curve detectors in these models, curve detectors are also found in monkeys. And we discover these high
low frequency detectors and then some follow up work went and discovered them in rats or mice. So they were found first in
artificial neural networks and then found in
biological neural networks. You know, there's this
really famous result on like grandmother neurons or the Halle Berry neuron
from Quiroga et al. And we found very similar
things in vision models where, this is while I was still at OpenAI and I was looking at their clip model, and you find these neurons that respond to the same entities in images. And also to give a concrete example there, we found that there was
a Donald Trump neuron. For some reason, I guess everyone likes to talk about Donald Trump, and Donald Trump was very prominent, was a very hot topic at that time. So every neural network we looked at, we would find a dedicated
neuron for Donald Trump. And that was the only person who had always had a dedicated neuron. You know, sometimes you'd
have an Obama neuron, sometimes you'd have a Clinton neuron, but Trump always had a dedicated neuron. So it responds to, you
know, pictures of his face and the word Trump, like
all these things, right? And so it's not responding
to a particular example or like, it's not just
responding to his face, it's extracting over this
general concept, right? So in any case, that's very similar to these Quiroga et al results. So there evidence that this
phenomenon of universality, the same things form
across both artificial and natural neural networks. That's a pretty amazing
thing if that's true. You know, it suggests that,
well, I think the thing that it suggests is
the gradient of descent is sort of finding, you
know, the right ways to cut things apart in some sense that many systems converge on, and many different neural networks' architectures converge on. That there's some
natural set of, you know, there's some set of abstractions
that are a very natural way to cut apart the problem, and that a lot of systems
are gonna converge on. That would be my kind of, you know, I don't know anything about neuroscience. This is just my kind of wild speculation from what we've seen. - Yeah, that would be beautiful
if it's sort of agnostic to the medium of the model that's used to form the representation. - Yeah, yeah, and, you know, it's a kind of a wild speculation based, you know, we only have some, a few data points that suggest this, but you know, it does seem like there's some sense in
which the same things form again and again in both, in certainly in natural neural networks and also artificially or in biology. - And the intuition behind
that would be that, you know, in order to be useful in
understanding the real world, you need all the same kind of stuff. - Yeah, well if we pick, I don't know, like the idea of a dog, right? Like, you know, there's
some sense in which the idea of a dog is like a natural category in the universe or
something like this, right? Like, you know, there's some reason, it's not just like a weird
quirk of like how humans factor, you know, think about the world that we have this concept of a dog. It's in some sense... Or like, if you have the idea of a line, like there's, you know,
like look around us, you know, there are lines, you know. It's sort of the simplest way to understand this room in some sense is to have the idea of a line. And so I think that that would be my instinct
for why this happens. - Yeah, you need a curved line, you know, to understand a circle, and you need all those shapes to understand bigger things and it's a hierarchy of
concepts that are formed, yeah. - And like maybe there are ways to go and describe, you know, images without reference
to those things, right? But they're not the simplest way or the most economical way
or something like this. And so systems converge
to these strategies would be my wild hypothesis. - Can you talk through
some of the building blocks that we've been referencing
of features and circuits? So I think you first described them in 2020 paper "Zoom In: An
Introduction to Circuits." - Absolutely, so maybe I'll start by just describing some phenomena, and then we can sort of build to the idea of features and circuits. - [Lex] Wonderful. - If you spent like quite a few years, maybe like five years to some extent with other things, studying
this one particular model Inception V1, which is
this one vision model. It was state of the art in 2015 and, you know, very much not
state of the art anymore. (Lex laughs) And it has, you know, maybe
about 10,000 neurons in it. And I spent a lot of time looking at the 10,000 neurons, odd
neurons of Inception V1. And one of the interesting
things is, you know, there are lots of neurons that don't have some obvious
interpretable meaning, but there's a lot of neurons and Inception V1 that do have really clean interpretable meanings. So you find neurons
that just really do seem to detect curves, and you find neurons that
really do seem to detect cars, and car wheels and car windows and, you know, floppy ears of dogs, and dogs with long snouts
facing to the right, and dogs with long snouts
facing to the left. And you know, different kinds of, there's sort of this whole
beautiful edge detectors, line detectors, color contrast detectors, these beautiful things we call
high/low frequency detectors. You know, I think looking at it, I sort of felt like a biologist, you know, you just, you're looking at this sort of new world of proteins, and you're discovering all these different proteins that interact. So one way you could try to understand these models
is in terms of neurons. You could try to be like, oh, you know, there's a dog detecting neuron and here's a car detecting neuron. And it turns out you can actually ask how those connect together. So you can go and say, oh, you know, I have this car detecting
neuron, how is it built? And it turns out in the previous layer, it's connected really
strongly to a window detector, and a wheel detector, and it's sort of car body detector. And it looks for the window above the car, and the wheels below, and the car chrome sort of in the middle, sort of everywhere but
especially on the lower part. And that's sort of a
recipe for a car, right? Like that is, you know, earlier we said that the thing we wanted from mech interp was to get algorithms to go and get, you know, ask what is
the algorithm that runs? Well, here we're just
looking at the weights of the neuron network and reading off this kind of recipe for detecting cars. It's a very simple crude
recipe, but it's there. And so we call that a
circuit, this connection. Well, okay, so the problem is that not all of the neurons are interpretable, and there's reason to think, we can get into this more later, that there's this
superposition hypothesis. There's reason to think that sometimes the right unit to analyze things in terms of is combinations of neurons. So sometimes it's not that
there's a single neuron that represents say a car,
but it actually turns out after you detect the car, the model sort of hides
a little bit of the car in the following layer and a bunch of dog detectors. Why is it doing that? Well, you know, maybe it just doesn't wanna do that much work
on cars at that point and you know, it's sort
of storing it away to go. So it turns out then this
sort of subtle pattern of, you know, there's all these neurons that you think are dog detectors and maybe they're primarily that, but they all a little bit contribute to representing a car in that next layer. Okay, so now we can't really think, there might still be something that, I don't know, you could
call it like a car concept or something, but it no longer
corresponds to a neuron. So we need some term for these
kind of neuron like entities, these things that we sort of would've liked the neurons to be, these idealized neurons, the things that are the nice neurons, but also maybe there's
more of them somehow hidden and we call those features. - And then what are circuits? - So circuits are these
connections of features, right? So, when we have the car detector and it's connected to a window detector, and a wheel detector, and it looks for the wheels below and the windows on top, that's a circuit. So circuits are just
collections of features connected by weights and
they implement algorithms. So they tell us, you know, how are features used? How are they built? How do they connect together? So maybe it's worth trying to pin down like what really is the
core hypothesis here. And I think the core
hypothesis is something we call the linear representation hypothesis. So if we think about the
car detector, you know, the more it fires, the more
we sort of think of that as meaning, oh, the model is more and more confident that a car is present. Or you know, if it's some
combination of neurons that represent a car, you know, the more that combination fires, the more we think the model
thinks there's a car present. This doesn't have to be the case, right? Like you could imagine something
where you have, you know, you have this car detector neuron and you think, ah, you know,
if it fires like, you know, between one and two, that means one thing, but it means like totally different if it's between three and four. That would be a nonlinear representation. And in principle that, you
know, models could do that. I think it's sort of
inefficient for them to do, if you try to think about how you'd implement computation like that, it's kind of an annoying thing to do. But in principle, models can do that. So one way to think about the features and circuits sort of framework
for thinking about things is that we're thinking about
things as being linear. We're thinking about there as being that if a neuron or a
combination neurons fires more, it's sort of, that means more of a particular thing being detected. And then that gives weights
a very clean interpretation as these edges between these entities, these features and that
edge then has a meaning. So that's in some ways the core thing. It's like, you know, we can talk about this sort of outset,
the context of neurons. Are you familiar with
the Word2Vec results? So you have like, you know, king minus man plus woman equals queen. Well, the reason you can
do that kind of arithmetic is because you have a
linear representation. - Can you actually explain that
representation a little bit? So, first of all, so the feature is a direction of activation. - Yeah, exactly.
- You can do it that way. Can you do the minus men plus women, that, the Word2Vec stuff, can you explain what that is that work? - [Chris] Yeah, so there's this very- - It's such a simple, clean explanation of what we're talking about. - Exactly, yeah. So there's this very famous result, Word2Vec by Tomas Mikolov et al, and there's been tons of
follow-up work exploring this. So, sometimes we have these, we create these word embeddings where we map every word to a vector. I mean, that in itself, by the way, is kind of a crazy thing if you haven't thought
about it before, right? Like we are going in and representing, we're turning, you know, like if you just learned about vectors in physics class, right? And I'm like, oh, I'm gonna actually turn every word in the
dictionary into a vector. That's kind of a crazy idea, okay. But you could imagine, you could imagine all kinds of ways in which you
might map words to vectors. But it seems like when
we train neural networks, they like to go and map words to vectors to such that they're
sort of linear structure in a particular sense, which is that directions have meaning. So for instance, there
will be some direction that seems to sort of
correspond to gender, and male words will be, you
know, far in one direction and female words will
be in another direction. And the linear
representation hypothesis is, you could sort of think of it roughly as saying that that's actually kind of the fundamental
thing that's going on, that everything is just
different directions have meanings and adding
direction vectors together can represent concepts. And the Mikolov paper sort
of took that idea seriously, and one consequence of it is that you can do this game of playing sort of arithmetic with words. So you can do king and you can, you know, subtract off the word man
and add the word woman. And so you're sort of, you know, going and trying to switch the gender. And indeed if you do that, the result will sort of be close to the word queen. And you can, you know, do other things like you can do, you know, sushi minus Japan plus Italy and get pizza or different things like this, right? So, this is in some sense the core of the linear
representation hypothesis. You can describe it just as a purely abstract thing. But vector spaces, you can
describe it as a statement about the activations of neurons. But it's really about this property of directions having meaning. And in some ways it's
even a little subtle that it's really I think
mostly about this property of being able to add things together that you can sort of independently modify say gender and royalty or, you know, cuisine type or country, and the concept of food by adding them. - Do you think the
linear hypothesis holds- - Yes.
- That kind of carries scales. - So, so far, I think
everything I have seen is consistent with the hypothesis, and it doesn't have to be that way, right? Like you can write down neural networks where you write weights such that they don't have linear representations, where the right way to understand them is not in terms of linear representations. But I think every natural neural network I've seen has this property. There's been one paper recently that there's been some sort
of pushing around the edge. So I think there's been some work recently studying
multi-dimensional features where rather than a single direction, it's more like a manifold of directions. This to me still seems like
a linear representation. And then there's been some other papers suggesting that maybe
in very small models, you get non-linear representations. I think that the jury's still out on that. But I think everything that we've seen so far has been consistent with the linear representation
hypothesis and that's wild. It doesn't have to be that way, and yet I think that
there's a lot of evidence that certainly at least this
is very, very widespread, and so far the evidence
is consistent with it. And I think, you know,
one thing you might say is you might say, well, Christopher, you know, that's a lot, you know, to go and sort of to ride on. You know, if we don't know
for sure this is true, and you're sort of, you know, you're investing in neural networks as though it is true, you
know, isn't that dangerous? Well, you know, but I think actually, there's a virtue in taking
hypotheses seriously and pushing them as far as they can go. So it might be that someday
we discover something that isn't consistent with
linear representation hypothesis. But science is full of hypotheses and theories that were wrong, and we learned a lot by
sort of working under them as a sort of an assumption, and then going and pushing
them as far as we can. I guess this is sort of the heart of what Kuhn would call normal science. I dunno if you want, we
can talk a lot about- - Kuhn.
- Philosophy of science and- - That leads to the paradigm shift. So yeah, I love it, taking
the hypothesis seriously, and take it to a natural conclusion. Same with the Scaling Hypothesis, same- - Exactly. Exactly.
- I love it. - One of my colleagues, Tom Henighan, who is a former physicist, like made this really nice analogy to me of caloric theory where, you know, once upon a time we thought
that heat was actually, you know, this thing called caloric, and like the reason,
you know, hot objects, you know, would warm up, cool objects is like the
caloric is flowing through them. And like, you know, because we're so used to thinking about heat, you know, in terms of the modern theory, you know, that seems kind of silly. But it's actually very hard
to construct an experiment that sort of disproves
the caloric hypothesis. And, you know, you can actually do a lot of really useful
work believing in caloric. For example, it turns out that the original combustion
engines were developed by people who believed
in the caloric theory. So I think there's a virtue in taking hypotheses seriously, even when they might be wrong. - Yeah, there's a deep
philosophical truth to that. That's kind of like how I
feel about space travel, like colonizing Mars. There's a lot of people
that criticize that. I think if you just assume
we have to colonize Mars in order to have a backup
for human civilization, even if that's not true,
that's gonna produce some interesting engineering and even scientific
breakthroughs, I think. - Yeah, well, and actually
this is another thing that I think is really interesting. So, you know, there's a way in which I think it can be really useful for society to have people
almost irrationally dedicated to investigating particular hypotheses because, well, it takes a lot to sort of maintain scientific morale and really push on
something when, you know, most scientific hypotheses
end up being wrong. You know, a lot of
science doesn't work out. And yet it's, you know, it's very useful to just, you know, there's a joke about Jeff Hinton, which is that Jeff Hinton has discovered how the brain works every
year for the last 50 years. But you know, I say that
with like, you know, with really deep respect because in fact that's actually, you know, that led to him doing some
some really great work. - Yeah, he won the Nobel Prize. Now who's laughing now? - [Chris] Exactly, exactly, exactly. - Yeah. - I think one wants to be able to pop up and sort of recognize the
appropriate level of confidence. But I think there's also a lot of value and just being like, you know, I'm going to essentially assume, I'm gonna condition on this problem being possible or this being
broadly the right approach, and I'm just gonna go and
assume that for a while and go and work within that
and push really hard on it. And, you know, society has lots of people doing that for different things. That's actually really useful in terms of going and getting to, you know, either really
ruling things out, right? We can be like, well, you
know, that didn't work and we know that somebody tried hard. Or going in and getting to something that it does teach us
something about the world. - So another interesting hypothesis is the superposition hypothesis. Can you describe what superposition is? - Yeah, so earlier we were
talking about word defect, right? And we were talking about how, you know, maybe you have one direction
that corresponds to gender, and maybe another that
corresponds to royalty, and another one that corresponds to Italy, and another one that
corresponds to, you know, food and all of these things. Well, you know, oftentimes
maybe these word embedding, they might be 500
dimensions, 1000 dimensions. And so if you believe that all of those directions were orthogonal, then you could only have,
you know, 500 concepts. And you know, I love pizza, but like, if I was gonna go and like give the like 500 most important concepts in, you know, the English language,
probably Italy wouldn't be, it's not obvious at least that Italy would be one of them, right? Because you have to have things like plural, and singular, and verb, and noun, and adjective, and, you know, there's a lot of things we have to get to before
we get to Italy, and Japan, and, you know, there's a lot
of countries in the world. And so how might it be that
models could, you know, simultaneously have the linear
representation hypothesis be true and also represent more things than they have directions. So, what does that mean? Well, okay, so if linear
representation hypothesis is true, something interesting has to be going on. Now, I'll tell you one
more interesting thing before we go and we do
that, which is, you know, earlier we were talking about all these polysemantic neurons, right? These neurons that, you know, when we were looking at Inception V1, there's these nice neurons
that like the car detector and the curve detector
and so on that respond to lots of, you know,
to very coherent things. But there's lots of neurons that respond to a bunch of unrelated things, and that's also an interesting phenomenon. And it turns out as well
that even these neurons that are really, really clean, if you look at the weak
activations, right? So if you look at like,
you know, the activations where it's like activating
5% of the, you know, of the maximum activation,
it's really not the core thing that it's expecting, right? So if you look at a curve
detector, for instance, and you look at the places
where it's 5% active, you know, you could interpret it just as noise or it could be that it's doing
something else there, okay? So, how could that be? Well, there's this amazing
thing in mathematics called compressed sensing, and it's actually this
very surprising fact where if you have a high dimensional space and you project it into
a low dimensional space, ordinarily you can't go
and sort of unproject it and get back your high
dimensional vector, right? You threw information away. This is like, you know, you can't invert a rectangular matrix, you can only invert square matrices. But it turns out that that's
actually not quite true. If I tell you that the high
dimensional vector was sparse, so it's mostly zeros, then it turns out that you can often go and find back the high dimensional vector with very high probability. So that's a surprising fact, right? It says that, you know, you can have this high
dimensional vector space, and as long as things are
sparse, you can project it down, you can have a lower
dimensional projection of it, and that works. So the superposition
hypothesis is saying that that's what's going on in neural networks. That's, for instance, that's what's going on in word embeddings. That word embeddings are able to simultaneously have directions
be the meaningful thing. And by exploiting the fact that they're operating on a fairly
high dimensional space, they're actually, and the fact that these
concepts are sparse, right? Like, you know, you usually aren't talking about Japan and Italy at the same time. You know, most of those
concepts, you know, in most instances, Japan
and Italy are both zero. They're not present at all. And if that's true, then you can go and have it be the case that you can have many more of
these sort of directions that are meaningful, these features than you have dimensions. And similarly, when we're
talking about neurons, you can have many more
concepts than you have neurons. So that's, at a high level,
the superposition hypothesis. Now it has this even wilder implication, which is to go and say
that neural networks are, it may not just be the case that the representations are like this, but the computation may
also be like this, you know, the connections between all of them. And so in some sense, neural
networks may be shadows of much larger, sparser neural networks, and what we see are these projection. And, you know, the strongest version of the superposition
hypothesis would be to take that really seriously and
sort of say, you know, there actually is in some
sense this upstairs model, this, you know, where the
neurons are really sparse and all interpretable,
and there's, you know, the weights between them are
these really sparse circuits. And that's what we're studying. And the thing that we're observing is the shadow of evidence. We need to find the original object. - And the process of learning
is trying to construct a compression of the upstairs model that doesn't lose too much
information in the projection. - Yeah, it's finding how
to fit it efficiently or something like this. The gradient descent is doing this. And in fact, so this sort of
says that gradient descent, you know, it could just represent a dense neural network, but it sort of says that gradient descent is implicitly searching over the space of extremely sparse models that could be projected into
this low dimensional space. And this large body of work of people going and trying to study sparse neural networks, right? Where you go and you have, you could design neural networks, right, where the edges are sparse and the activations are sparse. And you know, my sense is
that work has generally, it feels very principled, right? It makes so much sense. And yet that work hasn't really panned out that well is my impression broadly. And I think that a
potential answer for that is that actually the neural network is already sparse in some sense. Gradient descent was the whole time you were trying to go and do this, gradient descent was actually
in the behind the scenes going and searching more
efficiently than you could through the space of sparse models, and going and learning
whatever sparse model was most efficient and then figuring out how to fold it down nicely to go and run conveniently on your GPU, which does, you know, nice,
dense matrix multiplies, and that you just can't beat that. - How many concepts do you think can be shoved into a neural network? - Depends on how sparse they are. So there's probably an upper bound from the number of parameters, right? Because you have to have,
you still have to have, you know, weights that go
and connect them together. So that's one upper bound. There are in fact all these lovely results from compressed sensing and the Johnson-Lindenstrauss lemma, and things like this. That they basically tell you that if you have a vector space and you want to have
almost orthogonal vectors, which is sort of probably the thing that you want here, right? So you're gonna say, well, you know, I'm gonna give up on having my concepts, my features be strictly orthogonal, but I'd like them to
not interfere that much. I'm gonna have to ask them
to be almost orthogonal. Then this would say that
it's actually, you know, once you set a threshold for what you're willing to accept in terms of how much
cosine similarity there is, that's actually exponential in the number of neurons that you have. So at some point, that's not gonna even be the limiting factor. But there's some beautiful results there. And in fact, it's probably even better than that in some sense
because that's sort of, for saying that, you know, any random set of features could be active. But in fact the features have sort of a correlational
structure where some features, you know, are more likely to co-occur, and other ones are less
likely to co-occur. And so neural networks, my guess would be can do very well in terms of going and packing things in such, to the point that's probably
not the limiting factor. - How does the problem of polysemanticity enter the picture here? - Polysemanticity is this phenomenon we observe where you look at many neurons, and the neuron doesn't just
sort of represent one concept. It's not a clean feature. It responds to a bunch
of unrelated things. And superposition is, you can think of as being a hypothesis that explains the observation
of polysemanticity. So polysemanticity is
this observed phenomenon and superposition is a hypothesis that would explain it along
with some other things. - So that makes mech
interp more difficult. - Right, so if you're
trying to understand things in terms of individual neurons, and you have polysemantic neurons, you're in an awful lot of trouble, right? I mean, the easiest answer is
like, okay, well, you know, you're looking at the neurons, you're trying to understand them. This one responds for a lot of things, it doesn't have a nice meaning. Okay, you know, that's bad. Another thing you could ask is, you know, ultimately, we wanna
understand the weights. And if you have two polysemantic neurons and, you know, each one responds to three things and then, you know, the other neuron responds to three things and you have a weight between them, you know, what does that mean? Does it mean that like
all three, you know, like there's these nine, you know, nine interactions going on? It's a very weird thing. But there's also a deeper reason, which is related to the fact that neural networks operate on really high dimensional spaces. So I said that our goal was, you know, to understand neural networks and understand the mechanisms, and one thing you might
say is like, well, why not? It's just a mathematical function, why not just look at it, right? Like, you know, one of
the earliest projects I did studied these neural networks that match two dimensional spaces to two dimensional spaces, and you can sort of interpret them as in this beautiful way
as like bending manifolds. Why can't we do that? Well, you know, as you have
a higher dimensional space, the volume of that space in some senses is exponential in the
number of inputs you have. And so you can't just go and visualize it. So we somehow need to break that apart. We need to somehow break
that exponential space into a bunch of things that we, you know, some non-exponential number of things that we can reason about independently. And the independence is crucial because it's the
independence that allows you to not have to think about, you know, all the exponential
combinations of things. And things being mono-semantic, things only having one meaning. Things having a meaning, that is the key thing that allows you to think about them independently. And so I think that's, if
you want the deepest reason why we want to have interpretable
mono-sematic features, I think that's really the deep reason. - And so the goal here, as your recent work has been aiming at, is how do we extract the
mono-semantic features from a neural net that
has poly-sematic features and all this mess. - Yes, we observed these
poly-semantic neurons, and we hypothesized that's
what's going on is superposition. And if superposition is
what's going on there, there's actually a sort of
well established technique that is sort of the
principled thing to do, which is dictionary learning. And it turns out if you
do dictionary learning, in particular, if you do
sort of a nice efficient way that in some sense sort of
nicely regularizes it as well called a sparse auto-encoder. If you train a sparse auto-encoder, these beautiful interpretable features start to just fall out where
there weren't any beforehand. And so that's not a thing that you would necessarily predict, right? But it turns out that that
works very, very well. You know, that to me that
seems like, you know, some non-trivial validation of linear representations
in superposition. - So with dictionary
learning, you're not looking for particular kind of categories, you don't know what they are. They just emerge.
- Exactly. And this gets back to
our earlier point, right? When we're not making assumptions, gradient descent is smarter than us, so we're not making
assumptions about what's there. I mean, one certainly
could do that, right? One could assume that
there's a PHP feature and go and search for it,
but we're not doing that. We're saying we don't know
what's gonna be there. Instead we're just gonna go and let the sparse auto-encoder discover the things that are there. - So can you talk to the
"Toward Monosemanticity" paper from October last year? It had a lot of like nice
breakthrough results. - That's very kind of you
to describe it that way. Yeah, I mean, this was
our first real success using sparse auto-encoders. So we took a one layer model, and it turns out if you go and, you know, do dictionary learning on it, you find all these really
nice interpretable features. So, you know, the Arabic
feature, the Hebrew feature, the Base64 features,
those were some examples that we studied in a lot of depth, and really showed that they
were what we thought they were. It turns, if you train
a model twice as well and train two different models and do dictionary learning, you find analogous
features in both of them. So that's fun. You find all kinds of
of different features. So that was really just
showing that this work. And you know, I should mention that there was this Cunningham et al that had very similar
results around the same time. - There's something fun about doing these kinds of
small scale experiments and finding that it's actually working. - Yeah, well, and there's
so much structure here, like you know, so maybe
stepping back for a while, I thought that maybe all this mechanistic interpretability work, the end result was gonna be that I would have an explanation for why it was sort of, you know, very hard and not gonna be tractable. You know, we'd be like, well, there's this problem
with superposition, and it turns out
superposition is really hard, and we're kind of screwed,
but that's not what happened. In fact, a very natural,
simple technique just works. And so then that's actually
a very good situation. You know, I think this is a
sort of hard research problem and it's got a lot of research risk and you know, it might
still very well fail, but I think that some amount of, some very significant
amount of research risk was sort of put behind us
when that started to work. - Can you describe what kind of features can be extracted in this way? - Well, so it depends on the model that you're studying, right? So the larger the model, the more sophisticated they're gonna be, and we'll probably talk about
follow up work in a minute. But in these one layer models, so some very common things
I think were languages, both programming languages
and natural languages. There were a lot of features that were specific words in
specific contexts, so "the." And I think really the way to think about this is "the" is likely about to be followed by a noun. So it's really, you could
think of this as "the" feature, but you could also think of this as predicting a specific noun feature. And there would be these features that would fire for "the" in the context of say a legal document, or a mathematical document
or something like this. And so, you know, maybe
in the context of math you're like, you know, and
"the" then predict vector, matrix, you know, all
these mathematical words, whereas, you know, in other context, you would predict other
things, that was common. - And basically we need clever humans to assign labels to what we're seeing. - Yes, so, you know, this is, the only thing this is doing is that sort of unfolding things for you. So if everything was sort
of folded over top of it, you know, superposition folded
everything on top of itself and you can't really see
it, this is unfolding it. But now you still have
a very complex thing to try to understand. So then you have to do a bunch of work understanding what these are. And some of them are really subtle. Like there's some really cool things even in this one layer
model about Unicode where, you know, of course some
languages are in Unicode and the tokenizer won't necessarily have a dedicated token for
every Unicode character. So instead what you'll have is you'll have these patterns
of alternating token, or alternating tokens that each represent half
of a Unicode character. - Nice.
- And you'll have a different feature that, you know, goes and activates on the
opposing ones to be like, okay, you know, I just finished
a character, you know, go and predict next prefix. Then okay, on the prefix, you know, predict a reasonable suffix, and you have to alternate back and forth. So there's, you know,
these one player models are really interesting. And I mean, it's another thing
that just, you might think, okay, there would just
be one Base64 feature, but it turns out there's actually a bunch of Base64 features because you can have English
text encoded as Base64, and that has a very different distribution of Base64 tokens than regular. And there's some things about tokenization as well that it can exploit. And I dunno, there's all
all kinds of fun stuff. - How difficult is the task of sort of assigning
labels to what's going on? Can this be automated by AI? - Well, I think it depends on the feature and it also depends on how
much you trust your AI. So there's a lot of work doing
automated interpretability. I think that's a really
exciting direction, and we do a fair amount of
automated interpretability and have Claude go and label our features. - Is there some funny moments where it's totally right
or it's totally wrong? - Yeah, well, I think it's very common that it's like says
something very general, which is like true in some sense, but not really picking up on the specific of what's going on. So I think that's a
pretty common situation. Yeah, don't know that I have
a particularly amusing one. - That's interesting, that
little gap between it is true but doesn't quite get to
the deep nuance of a thing. That's a general challenge. It's like truly an
incredible accomplishment that it can say a true thing, but it doesn't, it's not, it's missing the depth sometimes. And in this context, it's
like the ARC challenge, you know, the sort of IQ type of tests. It feels like figuring out what a feature represents is a bit of, is a little puzzle you have to solve. - Yeah, and I think that
sometimes they're easier, and sometimes they're harder as well. So yeah, I think that's tricky. And there's another thing which, I dunno, maybe in some ways this is
my like aesthetic coming in, but I'll try to give
you a rationalization. You know, I'm actually a little suspicious of automated interpretability, and I think that partly just that I want humans to
understand neural networks, and if the neural network
is understanding it for me, you know, I don't quite like that. But I do have a bit of,
you know, in some ways, I'm sort of like the mathematicians
who are like, you know, if there's a computer automated proof, it doesn't count. - Right?
- You know, they won't understand it. But I do also think that there is this kind of like reflections on trusting trust type issue where, you know, if you, there's
this famous talk about, you know, like when you're
writing a computer program, you have to trust your compiler, and if there was like
malware in your compiler, then it could go and inject
malware into the next compiler, and you know, you'd be
kind of in trouble, right? Well, if you're using neural networks to go and verify that your
neural networks are safe, the hypothesis that you're
testing for is like, okay, well, the neural
network maybe isn't safe, and you have to worry about
like, is there some way that it could be screwing with you? So, you know, I think that's
not a big concern now, but I do wonder in the long run if we have to use really
powerful AI system to go and, you know, audit our AI systems, is that actually something we can trust? But maybe I'm just rationalizing 'cause I just want to us to have to get it to a point where humans
understand everything. - Yeah, I mean, especially
that's hilarious, especially as we talk about AI safety and it looking for features
that would be relevant to AI safety, like deception and so on. So let's talk about the
"Scaling Monosemanticity" paper in May, 2024. Okay, so what did it take to scale this, to apply to Claude 3s on it? - Well, a lot of GPUs. - A lot more GPUs, got it. - But one of my teammates, Tom Henighan was involved in
the original scaling laws work, and something that he
was sort of interested in from very early on is,
are there scaling laws for interpretability? And so something he
sort of immediately did when this work started to succeed, and we started to have
sparse auto-encoders work was he became very interested in, you know, what are the scaling laws for, you know, for making sparse
auto-encoders larger? And how does that relate to
making the base model larger? And so it turns out this works really well and you can use it to
sort of project, you know, if you train a sparse
auto-encoder at a given size, you know, how many tokens
should you train on? And so on. So this was actually a very big help to us in scaling up this work, and made it a lot easier
for us to go and train, you know, really large
sparse auto-encoders where, you know, it's not like
training the big models, but it's starting to get to a point where it's actually actually expensive to go and train the really big ones. - So you have this, I mean,
you have to do all this stuff of like splitting it across large GPUs- - Oh yeah, no, I mean there's a huge engineering challenge here too, right? So, yeah, so there's a
scientific question of, how do you scale things effectively? And then there's an enormous amount of engineering to go and scale this up. So you have to chart it, you have to think very
carefully about a lot of things. I'm lucky to work with a
bunch of great engineers 'cause I am definitely
not a great engineer. - Yeah, and the infrastructure especially, yeah, for sure. So it turns out, TLDR, it worked. - It worked, yeah. And I think this is important because you could have imagined, like, you could have imagined a world where you said after
towards mono-semanticity, you know, Chris, this is great, you know, it works on a one layer model, but one layer models are
really idiosyncratic. Like, you know, maybe,
that's just something, like maybe the linear
representation hypothesis and superposition hypothesis is the right way to
understand a one layer model, but it's not the right way
to understand larger models. And so I think, I mean, first of all, the Cunningham et al
paper sort of cut through that a little bit and sort of suggested that this wasn't the case. But scaling mono-semanticity sort of, I think was significant evidence that even for very large models, and we did it on Claude 3 Sonnet, which at that point was one
of our production models. You know, even these
models seemed to be very, you know, seemed to be
substantially explained at least by linear features. And, you know, doing dictionary
learning on them works, and as you learn more features, you go and you explain more and more. So that's, I think,
quite a promising sign. And you find now really
fascinating abstract features. And the features are also multimodal. They respond to images and text for the same concept, which is fun. - Yeah, can you explain that? I mean, like, you know, backdoor, there's just a lot of
examples that you can- - Yeah, so maybe let's start
with a one example to start, which is we found some features around sort of security vulnerabilities
and backdoors and codes. So it turns out those are
actually two different features. So there's a security
vulnerability feature, and if you force it
active, Claude will start to go and write security vulnerabilities like buffer overflows into code. And it also, it fires
for all kinds of things. Like, you know, some of the
top dataset examples for it were things like, you
know, dash dash disable, you know, SSL or something like this, which are sort of
obviously really insecure. - So at this point it's kind of like, maybe it's just because the examples were presented that way, it's kind of like a little bit
more obvious examples, right? I guess the idea is that down the line, it might be able to detect more nuanced, like deception or bugs
or that kind of stuff. - Yeah, well, I maybe wanna
distinguish two things. So one is the complexity of the feature or the concept, right? And the other is the nuance of how subtle the examples
we're looking at, right? So, when we show the top dataset examples, those are the most extreme examples that cause that feature to activate. And so it doesn't mean
that it doesn't fire for more subtle things. So, you know, the insecure code feature, you know, the stuff that it fires for, most strongly for are
these like really obvious, you know, disable the
security type things. But you know, it also fires
for, you know, buffer overflows and more subtle security
vulnerabilities in code. You know, these features
are all multimodal, so you could ask like, what
images activate this feature? And it turns out that the
security vulnerability feature activates for images of like people clicking on Chrome to
like go past the, like, you know, this website, the SSL certificate might be
wrong or something like this. Another thing that's very entertaining is there's backdoors and code feature. Like you activate it, it goes
and Claude writes a backdoor that like will go and dump
your data to port or something. But you can ask, okay, what images activate the backdoor feature? It was devices with
hidden cameras in them. So there's a whole
apparently genre of people going and selling devices
that look innocuous, that have hidden cameras and they have- - That's great.
- This hidden camera in it. And I guess that is the, you know, physical version of a backdoor. And so it sort of shows you how abstract these concepts are, right? And I just thought that was, I'm sort of sad that
there's a whole market of people selling devices like that, but I was kind of delighted that that was the thing that it came up with as the top
image examples for the feature. - Yeah, it's nice. It's multimodal. It's multi almost context. It's as broad, strong definition of a singular concept, it's nice. - Yeah. - To me, one of the really
interesting features, especially for AI safety
is deception and lying. And the possibility that
these kinds of methods could detect lying in a model, especially gets smarter
and smarter and smarter. Presumably that's a big threat
of a super intelligent model that it can deceive
the people operating it as to its intentions or
any of that kind of stuff. So what have you learned from detecting lying inside models? - Yeah, so I think we're in some ways in early days for that. We find quite a few features
related to deception and lying. There's one feature where, you know, fires for people lying
and being deceptive, and you force it active
and starts lying to you. So we have a deception feature. I mean, there's all
kinds of other features about withholding information and not answering questions, features about power seeking
and coups and stuff like that. So there's a lot of features that are kind of related to spooky things. And if you force them active, Claude will behave in ways that are, they're not the kinds
of behaviors you want. - What are possible
next exciting directions to you in the space of mech interp? - Well, there's a lot of things. So for one thing, I would
really like to get to a point where we have circuits where
we can really understand not just the features, but then use that to understand
the computation of models. That relief for me is the
ultimate goal of this. And there's been some work,
we put out a few things. There's a paper from Sam Marks that does some stuff like this. And there's been some, I'd say some work around the edges here. But I think there's a lot more to do, and I think that will be
a very exciting thing. That's related to a challenge
we call interference weights, where due to superposition, if you just sort of naively look at where their features
are connected together, there may be some weights that sort of don't exist in the upstairs model, but are just sort of
artifacts of superposition. So that's a sort of technical
challenge related to that. I think another exciting
direction is just, you know, you might think of sparse auto-encoders as being kind of like a telescope. They allow us to, you know, look out and see all these
features that are out there. And you know, as we build better and better sparse auto-encoders, get better and better
at dictionary learning, we see more and more stars, and you know, we zoom in on
smaller and smaller stars. But there's kind of a lot of evidence that we're only still seeing a very small fraction of the stars. There's a lot of matter in our, you know, neural network universe
that we can't observe yet. And it may be that we'll never be able to have fine enough
instruments to observe it, and maybe some of it just isn't possible, isn't computationally
tractable to observe it. So it's sort of a kind of dark matter, not in maybe the sense
of modern astronomy, but of early astronomy when we didn't know what this unexplained matter is. And so I think a lot
about that dark matter and whether we'll ever observe it, and what that means for safety if we can't observe it, if there's, you know, if
some significant fraction of neural networks are
not accessible to us. Another question that
I think a lot about is, at the end of the day, you know, mechanistic interpretability is this very microscopic approach
to interpretability. It's trying to understand things
in a very fine-grained way. But a lot of the questions we care about are very macroscopic. You know, we care about these questions about neural network behavior, and I think that's the thing
that I care most about, but there's lots of other sort of larger scale questions
you might care about. And somehow, you know, the nice thing about about having a very microscopic approach
is it's maybe easier to ask, you know, is this true? But the downside is it's much further from the things we care about, and so we now have this ladder to climb. And I think there's a question of, will we be able to find, are there sort of larger
scale abstractions that we can use to
understand neural networks? Can we get up from this
very microscopic approach? - Yeah, you've written about
this kind of organs question. - Yeah, exactly.
- So if we think of interpretability as a kind
of anatomy of neural networks, most of the circuits threads involve studying tiny little veins, looking at the small scale at individual neurons
and how they connect. However, there are many natural questions that the small scale
approach doesn't address. In contrast, the most
prominent abstractions in biological anatomy involve
larger scale structures, like individual organs, like the heart, or entire organ systems,
like the respiratory system. And so we wonder, is there a
respiratory system or heart or brain region of an
artificial neural network? - Yeah, exactly. And I mean, like if you
think about science, right? A lot of scientific fields have, you know, investigate things at many
levels of abstractions. In biology you have like, you know, molecular biology studying, you know, proteins and molecules and so on. And they have cellular biology, and then you have
histology studying tissues, and you have anatomy, and
then you have zoology, and then you have ecology. And so you have many,
many levels of abstraction or you know, physics, maybe the physics of individual particles, and then, you know, statistical physics gives you thermodynamics
and things like this. And so you often have different
levels of abstraction. And I think that right
now we have, you know, mechanistic interpretability
if it succeeds is sort of like a microbiology
of neural networks, but we want something more like anatomy. And so, and you know, a
question you might ask is, why can't you just go there directly? And I think the answer is superposition, at least in significant part. It's that it's actually very hard to see this macroscopic structure without first sort of breaking down the microscopic structure
in the right way, and then studying how
it connects together. But I'm hopeful that there
is gonna be something much larger than features and circuits, and that we're gonna
be able to have a story that involves much bigger things, and then you can sort of study in detail the parts you care about. - I suppose to neurobiology,
like a psychologist or a psychiatrist of a neural network. - And I think that the beautiful thing would be if we could go, and rather than having disparate fields for those two things, if you could build a bridge between them- - Oh, right.
- Such that you could go and have all of your
higher level abstractions be grounded very firmly
in this very solid, you know, more rigorous,
ideally, foundation. - What do you think is the difference between the human brain, the
biological neural network and the artificial neural network? - Well, the neuroscientists
have a much harder job than us. You know, sometimes I just
like count my blessings by how much easier my job is than the neuroscientists, right? So I have, we can record
from all the neurons. We can do that on
arbitrary amounts of data. The neurons don't change while you're doing that, by the way. You can go and ablate neurons, you can edit the connections and so on, and then you can undo those changes. That's pretty great. You can force, you can
intervene on any neuron and force it active and see what happens. You know, which neurons are
connected to everything, right? Neuroscientists wanna get the connectome, we have the connectome and we have it for like much bigger than like C. elegans. And then not only do
we have the connectome, we know what, you know, which neurons excite or inhibit each other, right. So we have, it's not just that we know that like the binary
mass, we know the weights. We can take gradients, we know computationally
what each neuron does. So I don't know the list goes on and on. We just have so many advantages
over neuroscientists. And then despite having
all those advantages, it's really hard. And so one thing I do
sometimes think is like, gosh, like if it's this hard for us, it seems impossible under the constraints of neuroscience or, you
know, near impossible. I don't know, maybe part of me is like, I've got a few neuroscientists on my team. Maybe I'm sort sort of like, ah, you know, maybe the neuroscientists,
maybe some of them would like to have an easier
problem that's still very hard and they could come and
work on neural networks. And then after we figure out things in sort of the easy little pond of trying to understand neural networks,
which is still very hard, then we could go back to
biological neuroscience. - I love what you've
written about the goal of mech interp research as two goals, safety and beauty. So can you talk about the
beauty side of things? - Yeah, so, you know,
there's this funny thing where I think some people want, some people are kind of
disappointed by neural networks, I think, where they're like,
ah, you know, neural networks, it's these just these simple rules, and then you just like
do a bunch of engineering to scale it up and it works really well. And like, where's the like complex ideas? You know, this isn't like a very nice, beautiful scientific result. And I sometimes think
when people say that, I picture them being like, you
know, evolution is so boring. It's just a bunch of simple rules and you run evolution for a
long time and you get biology. Like what a sucky, you know, way for biology to have turned out. Where's the complex rules? But the beauty is that the
simplicity generates complexity. You know, biology has these simple rules and it gives rise to,
you know, all the life and ecosystems that we see around us, all the beauty of nature, that
all just comes from evolution and from something very simple evolution. And similarly, I think
that neural networks build, you know, create enormous complexity and beauty inside and
structure inside themselves that people generally don't look at and don't try to understand because it's hard to understand. But I think that there is
an incredibly rich structure to be discovered inside neural networks, a lot of very deep beauty if we're just willing to take the time to go and see it and understand it. - Yeah, I love mech interp, the feeling like we are understanding or getting glimpses of understanding the magic that's going on
inside is really wonderful. - It feels to me like one of the questions that's just calling out to be asked, and I'm sort of, I mean, a lot of people are thinking about this,
but I'm often surprised that not more are is, how is it that we don't know how to
create computer systems that can do these things, and yet we have these amazing systems that we don't know how to
directly create computer programs that can do these things, but these neural networks can
do all these amazing things? And it just feels like that
is obviously the question that sort of is calling out
to be answered if you are, if you have any degree of curiosity. It's like how is it that
humanity now has these artifacts that can do these things
that we don't know how to do? - Yeah, I love the image of the circuits reaching towards the light of the objective function. - Yeah, it's just, it's this organic thing that we've grown and we have
no idea what we've grown. Well, thank you for working on safety and thank you for appreciating the beauty of the things you discover. And thank you for talking today, Chris. This was wonderful.
- Yeah. Thank you for taking the
time to chat as well. - Thanks for listening to this
conversation with Chris Olah, and before that with Dario
Amodei and Amanda Askell. To support this podcast, please check out our
sponsors in the description. And now let me leave you with
some words from Alan Watts. "The only way to make sense out of change is to plunge into it, move with it, and join the dance." Thank you for listening and
hope to see you next time.