Anca Dragan: Human-Robot Interaction and Reward Engineering | Lex Fridman Podcast #81
the following is a conversation with ANCA Jorgen a professor of Berkeley working on human robot interaction algorithms and looked beyond the robots function in isolation and generate robot behavior that accounts for interaction and coordination with human beings she also consults at way Moe the autonomous vehicle company but in this conversation she's 100% wearing her Berkeley hat she is one of the most brilliant and fun roboticists in the world to talk with I had a tough and crazy day leading up to this conversation so I was a bit tired even more so than usual but almost immediately as she walked in her energy passion and excitement for human robot interaction was contagious so I had a lot of fun and really enjoy this conversation this is the artificial intelligence podcast if you enjoy it subscribe I need to review it with five stars in the Apple podcast supported on patreon or simply connect with me on Twitter Alex Friedman spelled Fri DM a.m. as usual I'll do one or two minutes of as now and never any ads in the middle that can break the flow of the conversation I hope that works for you and doesn't hurt the listening experience this show is presented by cash app the number one finance app in the App Store when you get it use code Lex podcast cash app lets you send money to friends buy Bitcoin and invest in the stock market with as little as one dollar since cash app does fractional share trading let me mention that the order execution algorithm that works behind the scenes to create the abstraction of fractional orders is an algorithmic marvel so big props to the cash app engineers for solving a hard problem that in the end provides an easy interface that takes a step up to the next layer of abstraction over the stock market making trading more accessible for new investors and diversification much easier so again if you get cash out from the App Store or Google Play and use the code lex podcast you get $10 and cash app will also donate $10 the first an organization that is helping to advanced robotics and STEM education for young people around the world and now here's my conversation with Enka Droog on when did you first fall in love with robotics I think it was a very gradual process and it was somewhat accidental actually because I first started getting into programming when I was a kid and then into Mass and then into compute I disliked computer science was the thing I was gonna do and then in college I got into AI and then I applied to the robotics Institute at Carnegie Mellon and I was coming from this little school and Germany didn't know any nobody had heard of but I had spent an exchange semester at Carnegie Mellon so I had letters from Carnegie Mellon so that was the only play you know I might he said no Berkeley said no Stanford said no that was the only place I got into so I went there it's a robotics Institute and I thought that robotics is a really cool way to actually apply the stuff that I knew and loved like optimization so that's how I got into robotics I have a better story how I got into cars which is I you know I used to do mostly manipulation in my PhD but now I do kind of a bit of everything application wise including cars and I got into cars because I was here in Berkeley while I was a PhD student still for RSS 2014 better be organized in and he arranged for it was Google at a time to give us rides and self-driving cars and I was in a robot and it was just making decision after decision the right call and he was so amazing so it was a whole different experience right it's just I mean manipulation is so hard you can't do anything and there was was it the most magical robot you've ever met so like for me to mean Google self-driving car for the first time was like a transformative moment the guy had two moments like that that and spot mini I don't know if you met Bob many for Boston Dynamics I felt like I felt like I fell in love or something like it because I thought I know how a spot many works right it's just I mean there's nothing truly special it is it's great engineering work but the anthropomorphism that went on into my brain they came to life like a head little arm and like and looked at me he she looked at me you know I don't know there's a magical connection there and it made me realize wow robots can be so much more than things that manipulate objects they can be things that have a human connection Jeff was a self-driving car the moment like it was there a robot that truly sort of inspired you that was I remember that experience very viscerally riding in that car and being just wowed I I had the they gave us a sticker that said I wrote in a self-driving car and I had this cute little Firefly on yes and our logo that was like the smaller were like you had the really cute one yeah and and I put it on my laptop and I had that for years until I finally changed my laptop out and you know what about if we walk back you mention optimization at like what beautiful ideas inspired you in math computer science early on like why get into this field seems like a cold and boring field of math like what was exciting to you about it the thing is I liked math from very early on from fifth grade is when I got into the math Olympia and all of that are you competed yeah this it Romania is like our national sport do you speak I understand so I got into that fairly early and and it was little maybe to just theory with no kind of I didn't kind of how - didn't really have a goal and I didn't understanding which was cool I always liked learning and understanding but there was no can what am i applying this understanding to and so I think that's how I got into more heavily into computer science cuz it was it was kind of math meets something you can do tangibly in the world do you remember like the first program you've written okay the first program I've written with I kind of do it wasn't cute basic and fourth grade and it was drawing like a circle right yeah you know I don't know how to do that anymore right that's like the first thing that they taught me I was like you could take a special I wouldn't say was an extra isn't a sense an extracurricular so you could sign up for you know dance or music or programming and I did the programming thing and I was like what what I know why did you compete in program like these days Romania probably that's like a big thing there's a program of competition hmm what was that did that touch you at all did a little bit of the computer science Olympian but not not as seriously as I did the math Olympiad so is programming yeah it's basically here's a hard math problem solve it with a computer it was kind of yeah it's more like algorithms exactly it's not where's algorithmic so okay you kind of mentioned the Google self-driving car but outside of that Oh what's like who or what is your favorite robot real or fictional that I captivated your imagination throughout I mean I guess you kind of alluded to the Google self-driving the Firefly was a magical moment but is there something else it was I think there was the Lexus by the way this was back back then but yeah so good question I am okay my favorite fictional robot is Wally and I love how amazingly expressive it is some personal things a little bit about expressive motion kinds of things you were staying with you can do this and it's a head and it's a manipulator and what does it all mean I like to think about that stuff I love Pixar I love animation I love Wally has two big eyes I think or no yeah it has these um these cameras and they move so yeah that's it so you know it goes through and then it's super cute it's yeah I think you know the way it moves it's just so expressive the timing of that motion what is doing with its arms and what it's doing with these lenses is amazing and so I've I've really liked that from the start and then on top of that sometimes I shared this it's a personal story I share with people or when I teach about AI or whatnot my husband proposed to me by building a Wally and he actuated it so it's seven degrees of freedom including the lens thing and it kind of came in and it had the he made it have like a you know the belly box opening thing so it just did that and then it's filled out this box made out of Legos that open slowly and then BAM no yeah yeah it was it was quite quite it's at a bar it could be like the most impressive thing I've ever heard okay special connection to Wally long story short I like Wally because I like animation and I like robots and I like you know the fact that this was I we still have this robot to this day what how hard is that problem do you think of the expressivity of robots like the with the Boston Dynamics I never talked to those folks about this particular element I've talked to him a lot but it seems to be like almost an accidental side effect for them that they weren't I don't know if they're faking it they weren't trying to okay they do say that the the gripper on it was not intended to be a face I don't know if that's a honest statement but I think they're legitimate and so do we automatically just anthropomorphizing and youths up anything we could see about a robot it's like the the question is how hard is it to create a wall-e type robot that connects so deeply with us humans what do you think it's really hard right so it depends on what settings so if you want to do it in this very particular narrow setting where it does only one thing and it's expressive then you can get an animator you know can have fixer on call come in design some trajectory is there was a a key had a robot called Cosmo where they put in some of these animations that part is easy right the hard part is doing it not via these kind of handcrafted behaviors but doing it generally autonomously like I want robot say I don't work on just to clarify I don't I used to work a lot on this I don't work on that quite as much these days but but have the notion of having robots that you know when they pick something up and put it in a place they can do that with various forms of style or you can say well this robot is you know succeeding at this desk and is confident versus its hesitant versus you know maybe it's happy or it's you know disappointed about something some failure that it had or I think that when robots move they can communicate so much about internal states or perceived internal states that they have and I think that's really useful in an element that we'll want in the future because I was reading this article about how kids are kids are being rude to Alexa because they can be rude to it and it doesn't really get angry right it doesn't reply it in any way it just says the same thing so I think there's at least for that for the for the correct development of children to learn that these things and you kind of react differently I also think you know you walk in your home and you have a personal robot and if you're really pissed presumably robot just kind of behave slightly differently than one you're super happy and excited but it's really hard because it's I don't know I don't you know the way I would think about it and the way I've thought about it when it came to in expressing goals or intent its our intentions for robots it's well what's really happening is that instead of doing robotics where you have your state and you have your action space and you have your space the reward functions are trying to optimize now you kind of have to expand the notion of state to include this human internal state what is the person actually perceiving what do they think about the robots something's better and then you have to optimize in that system and so that means you have to understand how your motion your actions end up sort of influencing the observers kind of perception of you and it's very it's very hard to write math about that right so when you start to think about incorporating the human into the state model apologize for the philosophical question but how complicated are human beings do you think like can they be reduced to two kind of almost like an object that moves and maybe has some basic intents or is there something do we have to model things like mood and general aggressiveness and time I mean all these kinds of human qualities or like game theoretic qualities like what's your sense how complicated it is how hard is the problem of human robot interaction yeah should we talk about what the problem of human robot is yeah this is what I mean talk about how that yeah so and by the way I'm gonna talk about this very particular view of human robot interaction right which is not so much on the social side or on the side of how do you have a good conversation with the robot what should the robots appearance be throws out that if you make robots taller versus shorter this has an effect on how people act with them so I'm not I'm not talking about that but I'm talking about this very kind of narrow thing which is you take if you want to take a task that a robot can do in isolation in a lab out there in the world but in isolation and now you're asking what does it mean for the robot to be able to do this task for presumably what it's actually angola's which is to help some person that ends up changing the problem in two ways the first way to changes the problem is that the robot is no longer the single agent acting there you have humans who also take actions in that same space you know cars navigating around people robots around an office navigating around the people in that office if I send the robot to over there in the cafeteria to get me a coffee then there's from other people reaching for stuff in the same space and so now you have your robot and you're in charge of the actions that the robot is taking then you have these people who are also making decisions and taking actions in that same space and even if you know the robot knows what it's what it should do and all of that just coexisting with these people right kind of getting the since the gel well to mesh well together that sort of the problem number one and then there's problem number two which is goes back to this notion of I if I'm a programmer I can specify some objective for the robot to go off and optimize you can specify the task but if I put the robot in your home presumably you might have your own opinions about well okay I want my house clean but how do I want it clean then how should robot how close to me it should come and all of that and so I think those are the two differences that you have your acting around people and you what you should be optimizing for should satisfy the preferences of that end user not of your programmer who programmed you yeah and the Preferences thing is tricky so figuring out those preferences be able to interactively adjust to understand what the human is so really boys ought to be understand the humans in order to interact with them in order to please them right so why is this hard what yeah why is understanding humans hard so I think there's two tasks about understanding humans that in my mind are very very similar but not everyone agrees so there's the task of being able to just anticipate what people will do we all know that cards need to do this right we all know that well if I navigate around some people the robot has to get some notion of ok where where is this person gonna be so that's kind of the prediction side and then there's what what you are saying satisfying the preferences right so adapting to the person's preference is knowing what to optimize for which is more this inference side this what is what does this person want what is their intent what are their preferences and to me those kind of go together because I think that in if you at very least if you can understand if you look at human behavior and understand what it is that they want then that's sort of the key enabler to being able to anticipate what they'll do in the future because I think that you know we're not arbitrary we make these decisions that we make we act in the way we do because we're trying to achieve things and so I think that's the relationship between them now how complicated do these models need to be in order to be able to understand what people want so we've gotten a long way in robotics with something called inverse reinforcement learning which is the notion of someone acts demonstrates what how they want this thing done what isn't inverse reinforcement learning you said it right so it's it's the problem of take human behavior and infer reward function from this figure out what it is that that behavior is optimal with respect to and it's a great way to think about learning human preferences in the sense of you know you have a car and the person can drive it and then you can say well okay I can actually learn what the person is optimizing for I can learn their driving style or you can you can have people demonstrate how they want the house clean and then you can say okay this is this is I mean I'm getting the trade-offs that they're that they're making I'm getting the Preferences that they want out of this and so we've been successful in robotics somewhat with this and it's a it's based on a very simple model of human behavior which is remarkably simple which is that human behavior is optimal with respect to whatever it is that people want right so you make that assumption and now you can kind of inverse through that's why it's called inverse well really optimal control but but also inverse reinforcement learning so this is based on utility maximization in economics press back in the forties fine women mortgage time or like okay people are making choices by maximizing utility go and then in the late 50s we had loose and Shephard come in and say people are a little bit noisy and approximate in that process so they might choose something kind of stochastic lee with probability proportional to how much utility something has there's a bit of noise in there on this has translated into buttocks and something that we call Boltzmann rationality so it's a kind of an evolution of inversed reinforcement learning that accounts for four human noise and we've had some success with that too for these tasks where it turns out people act noisily enough that you can't just do vanilla the vanilla version ah you can account for noise and still infer what what they seem to want based on this man now we're hitting tasks word that's no not enough and what are examples where are you damn desk so imagine you're trying to control some robot that's that's fairly complicated trying to control the robot arm cuz maybe you're a patient with a motor impairment and you have this wheelchair mounted army in China to control it around or one test that we've looked at with Sergei is and our students did is a lunar lander so just I don't know if you know this Atari game it's called lunar lander it's it's really hard people really suck at landing the same mostly they just crash it left and right okay so this is the kind of toss for imagine you're trying to provide some assistance to a person operating such such a robot where you won the kind of the autonomy to kick can figure out what it is that you're trying to do and help you do it it's really hard to do that for say lunar lander because people are all over the place and so they seem much more noisy than really irrational that's an example of a task where these models are kind of failing us and it's not surprising because so we you know we talk about a 40s utility late fifties sort of noisy then the seventies came and behavioral economics started being a thing where people are like no no no no no people are not rational people are messy and emotional and irrational and have all sorts of heuristics that might be domain-specific and they're just they're just a messy mess so so what do you so what does my robot do to understand what you won and it's a very it's very that's why it's complicated it's you know for the most part we get away with pretty simple models until we don't and then the question is what do you do then um and it I had days when I wanted to you know pack my bags and go home and jobs because it's just it feels really daunting to make sense of human behavior enough that you can reliably understand what people want especially as you know robot capabilities will continue to get developed you'll get these systems that are more and more capable of all sorts of things and then you really want to make sure that you're telling them the right thing to do what is that thing well read it in human behavior so if I just sit here quietly and try to understand something about you but listening to you talk it would be harder than if I got to say something and ask you and interact and control okay can you can the robot help its understanding of the human by inflowing it influencing the behavior by actually acting yeah absolutely so one of the things that's been exciting to me lately is this notion that when you tried to that that that when you try to think of the robotics problem as okay I have a robot and it needs to optimize for whatever it is that a person wants it to optimize as opposed to maybe what a programmer said that problem we think of as a human robot collaboration problem in which both agents get to act in which the robot knows less than the human because the human actually has access to and you know at least implicitly to what it is that they want they can't write it down but they can they can talk about it they can give all sorts of signals they can demonstrate and and but the robot doesn't need to sit there and passively observe human behavior and try to make sense of it the robot can act too and so there's these information gathering actions that the robot can take to sort of solicit responses that are actually informative so for instance this is not for the purpose of assisting people but with kind of back to coordinating with people in cars and all of that one thing that dorsa did was so we were looking at cars being able to navigate around people and you might not know exactly the driving style of a particular individual that's next to you but you want to change lanes in front of them navigating around other humans inside cars yeah good good clarification question so you have an autonomous car and it's trying to navigate the road around human driven vehicles similar things ideas applied to pedestrians as well but let's just take human driven vehicles so now you're trying to change a lane well you could be trying to infer the driving style of this person next to you you'd like to know if they're in particular if they're sort of aggressive or defensive if they're gonna let you kind of go in or if they're gonna not and and it's very difficult to just you know when if you think that if you want to hedge your bets that maybe they're actually pretty aggressive I shouldn't ride this you kind of end up driving next to them and driving next to them right and then you you don't know because you're not actually getting the observations that you get away someone drives when they're next to you and they just need to go straight it's kind of the same because if they're aggressive or defensive and so you need to enable the robot the reason about how it might actually be able to gather information by changing the actions that it's taking and then the robot comes up with these cool things where it kind of not just towards you and then sees if you're gonna slow down or not then if you slow down it sort of updates its model of you and says oh okay you're more on the defensive side so now I can actually that's a fascinating dance as so that's so cool you could use your own actions to gather information that's uh that feels like I'm totally open exciting new world of robotics prop I mean how many people are even thinking about that kind of thing because it's it's actually leveraging human I mean most roboticist I've talked to a lot of you know colleagues and so on are kind of being honest kind of afraid of humans because they're messy and complicated right I understand um going back to what we're talking about earlier right now we're kind of in this dilemma okay there are tasks that we can just assume people are approximately rational for and we can figure out what they want we can figure out their goals in fear are their driving styles whatever cool they're these tasks that we can't so what do we do right do we pack our bags and go home and this one is just I've had a little bit of hope recently um and I'm kind of doubting myself scoff what do I know that you know 50 years of behavioral economics hasn't figured out but maybe it's not really in contradiction with what with the way that field is headed but basically one thing that we've been thinking about is instead of kind of giving up and saying people are too crazy and irrational for us to make sense of them maybe we can give them a bit the benefit of the doubt and maybe we can think of them as actually being relatively rational but just under different assumptions about the world about how the world works about you know they don't have we when we think about rationality and bliss the assumption is or they're rational under all the same assumptions and constraints as the robot right what if this is the state of the world that's what they know this is the transition function that's what they know this is the horizon that's what they know but maybe maybe the kind of this difference the way the reason they can seem a little messy and hectic especially to robots is that perhaps they just make different assumptions or have different beliefs so I mean that's that's another fascinating idea that this are kind of anecdotal desire to say that humans are irrational perhaps grounded behavioral economics is is that we just don't understand the constraints and their awards under which they operate and so our goal shouldn't be to throw our hands up and say they're irrational is to say let's try to understand what are the constraints what it is that there must be assuming that makes this behavior make sense good life lesson right good life that's true it's just outside a robot is good too that's communicating with humans that's just a good assume that you just don't have empathy right it's uh this is maybe there something you're missing and you know and it's you know it especially happens to robots because they're kind of dumb and they don't know things and oftentimes people are sort of super irrational and that they actually know a lot of things that robots don't sometimes like with the lunar lander the robot you know knows much more so it turns out that if you try to say look maybe people are operating this thing but assuming a much more simple fight physics model because they don't get the complexity of this kind of craft or the robot arm with seven degrees of freedom when these inertia and whatever so so maybe they have this intuitive physics model which is not you know this notion of intuitive physics is something that good you just studied actually in cognitive science was like Josh Tenenbaum Tom Griffiths what kind of stuff and and what we found is that you can actually try to figure out what what physics model kind of best explains human actions and then you can use that to sort of correct what it is that they're commanding the craft to do so they might you know be sending the craft somewhere but instead of executing that action you can sort of take a step back and say according to their intuitive if the world worked according their intuitive physics model where do they think that the craft is going war day where are they trying to send it to and then you can use the real physics right the universe of that to actually figure out what you should do so that you do that instead of where they were actually sending you in the real world and I kid you not it word peopled landed there the damn thing and you know in between the two flags and and and all that so it's not conclusive in any way but I'd say it's evidence that yeah maybe we're kind of under estimating humans in some ways when we're giving up and saying oh there's just crazy noisy then you then you try to explicitly try to model the kind of worldview that they that they have that's right that's right it's not to I mean there's things to be here for Konami's through that that that for instance I've touched upon the planning horizon so there's this idea that I just bounded rationality essentially and the idea that well maybe we work under computational constraints and I think kind of our view recently has been take the bellmen update nai and just break it in all sorts of ways by saying state no no no the person doesn't get to see the real state maybe they're estimating somehow transition function no no no no even the actual reward evaluation maybe they're still learning about what it is that they want like like you know when you watch netflix and you know you have all the things and then you have to pick something imagine that you know the D the AI system interpreted that choice as this is the thing you prefer to see and how are you gonna know you're still trying to figure out what you like what you don't like etc so I mean it's important to also account for that so it's not irrationality precise doing the right thing under the things that they know yeah that's brilliant you mentioned recommender systems what kind of and we're talking about human robot interaction kind of problem spaces are you thinking about so is it robots like wheeled robots of autonomous vehicles is it object manipulation like when you think about human robot interaction in your mind and maybe I'm tree could speak for the entire community of human robot interaction no but like what are the problems of interest here is and does it you know I kind of think of open domain dialogue as human robot interaction and that happens not in the physical space but it could just happen in in the virtual space so word who wears the boundaries of this field for you when you're thinking about the things we've been talking about yeah so I I tried to find kind of underlying I don't know what to even call them I get try to work on you know I might call what I do the kind of working on the foundations of algorithmic human robot interaction and trying to make contributions there and and it's important to me that whatever we do is actually somewhat domain agnostic when it comes to is it about you know autonomous cars or is it about quadrotors or is it a basis or the same underlying principles apply of course when you're trying to get a particular to work usually have to do some extra work to adapt that to that particular domain but these things that we were talking about around well you know how do you model humans it turns out that a lot of systems need to quote benefit from a better understanding of how human behavior relates to what people want and need to predict human behavior physical robots of all sorts and and beyond that and so I used to do manipulation I used to be you know picking up stuff and then I was picking up stuff with people around and now it's sort of very broad when it comes to the application level but in a sense very focused on ok how does the problem need to change how do the algorithms need to change when we're not doing a robot by itself you know emptying the dishwasher but we're stepping outside of that oh I thought that popped into my head just now on the game theoretic side I think you said this really interesting idea of using actions to gain more information but if we think a sort of game theory the humans that are interacting with you with you the robot identity of the robot yeah is they also have a world model of you mm-hmm and you can manipulate that and if we look at autonomous vehicles people have a certain viewpoint you said with the kids people see Alexa as a in a certain way is there some value in trying to also optimize how people see you as a robot is that it or is that a little too far and away from the specifics of what we can solve right now so both right so it's really interesting and we've seen a little bit of progress on this problem on pieces of this problem so you can again it kind of comes down to how complicated is the human model need to be but in one piece of work that we were looking at we just said ok there's these in there's this that are internal to the robot and their what their what the robot is about to do or maybe what objective what driving style the robot has or something like that and what we're gonna do is we're going to set up a system where part of the state is the person's belief over those parameters and now when the robot acts that the person gets new evidence about this robot internal state and so they're updating their mental model of the robot right so if they see a card that sort of cut someone off Tory god that's an aggressive card they no more right if they see sort of a robot head towards a particular door they're like are the robots trying to get to that door so this thing that we have to do with humans to try to understand their goals and intentions humans are inevitably gonna do that to robots and then that raises this interesting question that you asked which is can we do something about that this is gonna happen inevitably but we can sort of be more confusing or less confusing to people and it turns out you can optimize for being more informative and less confusing if you if you have an understanding of how your actions are being interpreted by the human how they're using these actions to update their belief and honesty all we did is just Bayes rule basically okay first has a belief they see an action they make some assumptions about how the robot generates its actions presumably is being rational because robots are rational see reasonable to assume that about them and then they incorporate that that new piece of evidence the Bayesian sense and their belief and they obtain a posterior and now the robot is trying to figure out what actions to take such that it steers the person's belief to put as much probability mass as possible on the correct on the correct parameters so that's kind of a mathematical formalization of that but my worry and I don't know if you want to go there with me but I about this quite a bit um the the kids talking to alexa disrespectfully worries me i worry in general about human nature I guess I grew up in Soviet Union World War two I'm gonna do two so with the Holocaust and everything I just worry about how we sometimes treat the other the the group that we call out or whatever it is through human history the group that's the other has been changed faces but it seems like the robot will be the other the other the the next the other and one thing is it feels to me that robots don't get no respect they get shoved around shoved around in is there one at the shallow level for a better experience it seems that robots need to talk back a little bit like into my intuition says I mean most companies from sort of Roomba autonomous vehicle companies might not be so happy with the idea that a robot has a little bit of an attitude but I feel it feels to me that that's necessary to create a compelling experience like we humans don't seem to respect anything that doesn't give us some attitude that or like Miss mix of mystery and attitude and anger and did that threatens us subtly maybe passive-aggressively I don't it seems like we humans yet need that dude what are you is there something you have thoughts on this one is one is it it's we respond to you know someone being assertive but we also respond to someone being vulnerable so I think robots but my first thought is that robots get shoved around and and bullied a lot because they're sort of you know tempting and they're so showing off or they appear to be showing off and so I think current going back to these things we were talking about in the beginning of making robots a little more a little more expressive a little bit more like oh that wasn't cool to do and now I'm bummed right I think that that can actually help because people can't help but anthropomorphize and respond to that even that though the emotion being communicate is not in any way a real thing and people know that it's not a real T because they know it's just a machine we're still interpret you know we can work with we watch there's this a famous psychology experiment with little triangles and kind of dots on a screen and a triangle is chasing the square and get angry at the darn triangle because why is it not leaving the square alone so that's yeah we can't helps that was the first thought the vulnerability is really interesting that I I think of like being pushing back being assertive as the only mechanism of getting of forming a connection of gaining respect but perhaps vulnerability perhaps there's other mechanisms that are less threatening yeah a little bit yes but then this this other thing that we can think about is it goes back to what you were saying that interaction is really game theoretic all right so the moment you're taking actions in the space humans are taking actions in that same space but you have your own objective which is you know you're a car you need to get your passenger to the destination and then the human nearby has their own objective which someone overlaps with you but not entirely you boat you're not interested in getting into an accident with each other but you have different destinations and you want to get home faster and they want to get home faster and that's a general of some game at that point and so that's I think that's what it's reading it as such is kind of a way we can step outside of this kind of mode that where you try to anticipate what people do and you don't realize you have any influence over it while still protecting yourself because your understanding that people also understand that they can influence you and it's just kind of back and forth is this negotiation which is really really talking about different equilibria of a game the very basic way to solve coordination is to just make predictions about what people will do and then stay out of their way and that's hard for the reasons we talked about which is how you have to understand people's intentions implicitly explicitly who knows but somehow you have to get enough of an understanding of that we all anticipate what happens next and so that's challenging but then it's further challenged by the fact that people change what they're do based on what you do because they don't they don't plan in isolation either right so when you see cars trying to merge on a highway and not succeeding one of the reasons this can be is because you you they they look at traffic that keeps coming they predict what these people are planning on doing which is to just keep going and then they stay out of the way because there's not there's no feasible plan right any planning would actually intersect with one of these other people so that's bad so you get stuck there so now kind of if if you start thinking about it as no no no actually these people change what they do depending on what the car does like if the car actually tries to kind of inch itself forward they might actually slow down and let the car in and down take an advantage of that well that you know that's kind of the next level we call this like this under actuated system idea where it's gonna under actresses and robotics but it's kind of it's you don't your influence these other degrees of freedom but you don't get to decide what somewhere it's seen you mention it this the the human element in this picture as under actuate it said you know you understand under actuator about robotics is you know that you can't fully control the system so you can't go in arbitrary directions in the configuration space under your control yeah it's a very simple way of under actuation where basically there's literally these degrees of freedom that you can control and these are affirmed that you can't but you influence them and I think that's the important part is that they don't do whatever regardless of what you do that what you do influence is what they end up doing I just also like the the poetry of calling human robot interaction and under actuated robotics problem and y'all so much sort of nudging it seems that there and I don't know I think about this a lot in the case of pedestrians I've collected hundreds of hours of videos I like to just watch pedestrians mmm-hmm and it seems that it's a funny hobby yeah it's weird because I learn a lot I learned a lot about myself about our human human behavior from watching pedestrians watching people in their environment basically crossing the street is you're putting your life on the line you know I don't know tens of millions of time in America every day is people are just like playing this weird game of chicken when they cross the street especially when there's some ambiguity about the right-of-way that has to do either with the rules of the road or with the general personality of the intersection based on the time of day and so on I mean and this nudging idea I don't you know it seems that people don't even nudge they just aggressively take make a decision somebody there's a runner that gave me this advice I sometimes run in in the street and you know not in this jannah sidewalk and you said that if you don't make eye contact with people when you're running they will all move out of your way it's called civil and attention civil inattention that's the thing oh wow I need to look this stuff but it works what is that my sense was if you communicate like confidence in your actions that you're unlikely to deviate from the action that you're following that's a really powerful signal to others that they need to plan around your actions as opposed to nudging where you're sort of hesitantly then the hesitation might communicate that you're now you're still in the dance in the game that they can influence with their own actions I've recently had conversation with Jim Keller who is a sort of this legendary chip or chip architect but he also let the autopilot in for a while and his intuition that driving is fundamentally still like a ballistics problem like you can ignore the human element that it's just not hitting things and you can kind of learn the right dynamics required to do the merger and all those kinds of things and then my sense is and I don't know if I can provide a definitive proof of this but my sense is I can order a magnitude or more more difficult when humans are involved like it's not simply a object a collision avoidance problem which where does your intuition of course nobody knows the right answer here but where does your intuition fall on the difficulty fundamental difficulty of the driving problem when humans are involved yeah good question I have many opinions on this imagine downtown San Francisco yeah yeah it's crazy busy everything okay now take all the humans out no pedestrians no human driven vehicles no cyclists no people and little skill electric scooters have been around nothing I think we're done I think driving at that point is done we're done I did nothing really that's nice tilt needs to be solved about that well let's pause there i I think I agree with you that guy and I think a lot of people here will agree with that but we need to sort of internalize that idea so what's the problem there because we're not quite yet be done with that because a lot of people kind of focus on the perception problem well a lot of people kind of map autonomous driving into how close are we to solving being able to detect all the you know the the drivable area the objects in the scene do you see that as a how hard is that problem so your intuition there behind your statement was we might have not solved the yet but were close to solving basically the perceptual problem I think the perception problem I mean and by the way a bunch of years ago this would not have been true and a lot of issues and the space can't we're coming from the fact that we don't really you know we don't know what's what's where but I think it's fairly safe to say that at this point although you could always improve on things and all of that you can drive through downtown San Francisco if there are no people around there's no really perception issue standing in your way there any perception is hard but yeah it's we've made a lot of progress on the perceptions on how to undermine the difficulty of the problem I think everything about robotics is really difficult of course you know the the planning problem the control problem all very difficult but I think what's what makes it really you know yeah it might be I mean you know and I picked downtown San Francisco I ate adapting to well now it's snowing now is no longer snowing now it's slippery in this way now so the dynamics part could good I could imagine being being still somewhat challenging but no the thing that I think worries us and our tuition is not good there is the perceptual problem at the edge cases sort of stout sauce and Francisco the nice thing it's not actually it may not be a good example because cuz you know what - what you're getting for all there's like because crazy construction zones and all yeah but the thing is you're travelling at slow speeds so it doesn't feel dangerous to me what feels dangerous is highway speeds when everything is to us humans super clear yeah I'm assuming light are here by the way I think it's kind of irresponsible to not use lighter that's just my personal opinion depending on your use case but I think like you know if you if you have the opportunity to use light are good your injury makes more sense now so you don't think vision I really just don't know enough to say well vision alone what you know what's like I there's a lot of how many cameras do you have there's all sorts of details I imagine their stuff is really hard to actually see how do you deal with would glare exactly what you're saying stuff that people would see that that that you don't I I think I have more my intuition comes from systems that can actually use lighter as well yeah until we know for sure it's make sense to be using lidar that's kind of the safety focus but then deserve the I also sympathize with the Elon Musk the statement of lidar as a crutch it's it's it's uh it's a fun notion to think that the things that work today is a crutch for the invention of the things that will work tomorrow right they get it's kind of true in the sense that if we you know that we want to stick to the conference and you see this in academic and settings all the time the things that work force you to not explore outside think outside the box I mean that happened all of that the problem is in safety critical systems you kind of want to stick with a thing Sutekh work so it's a it's an interesting and difficult trade-off in the in the in the case of real-world sort of safety critical robotic systems but so your intuition is just to clarify yes how I mean how hard is this human element forger like how hard is driving when this human element is involved are we years decades away from solving it but perhaps actually the years and the the thing I'm asking it doesn't matter what the timeline is but do you think we're how many breakthroughs away away from its in solving the human robot interaction problem to get this to get this right I think it in a sense it really depends I think that you know we were talking about how well look it's really hard because I'm just know people do is hard and on top of that playing the game is hard but I think we sort of have the fundamental some of the fundamental understanding for that and then you already see that these systems are being deployed in the real world you know even even driverless because I think now a few companies that don't have a driver in the car yeah small areas he's got a chance to I went to Phoenix and I and I shot a video with lame-o and you need to get that video out people didn't give me slack but this incredible engineering work being done there and it's one of those other seminal moments for me in my life to be able to it sounds silly but to be able to drive without a with a ride sorry without a driver in the seat I mean I was an incredible robotics I was driven by a robot and without being able to take over without being able to take the steering wheel that's a magical that's a magical moment so in that regard and those domains at least for like way mo they're there they're solving that human there's I mean there were they're going fattening it felt fast because you're like freaking out at first I was this is my first experience but it's going like the speed limit right 30 40 whatever it is and there's humans and it deals with them quite well I detects them and a good negotiation the intersections the left turns and all that so at least in those domains it's solving them the open question for me is like how quickly can we expand you know that's the you know outside of the weather conditions all those kinds of things how quickly can we expand to like cities like San Francisco yeah and I wouldn't say that it's just you know now it's just pure engineering and it's probably the I mean I know by the way I'm speaking kind of very generally here as hypothesizing but I I think that that there are successes and yet no one is everywhere out there so that seems to suggest that things can be expanded and can be scaled and we know how to do a lot of things but they're still probably you know new algorithms or modified algorithms that that you still need to put in there as you as you learn more and more about new challenges that get you get faced when how much is this problem do you think can be learned through in turn this is the success of machine learning and reinforcement learning how much of it can be learned from sort of data from scratch and how much which most of the success of autonomous vehicle systems have a lot of heuristics and rule based stuff on top like human expertise in in injected forced into the system to make it work hmm what's your what's your sense how much what's the will be the role of learning in the near term I think I I think on the one hand that learning is inevitable here right I think on the other hand and when people characterize the problem as it's a bunch of rules that some people wrote versus it's an end-to-end RL system or imitation learning then maybe there's kind of something missing from maybe that's that's more so for instance I think a very very useful tool in this sort of problem both in how to generate the cars behavior and robots in general and how to model human beings is actually planning search optimization right so robotics is a Disick Winchell decision-making problem and when when a robot can figure out on its own how to achieve its goal without hitting stuff and all that stuff you're right all the good stuff promotion planning 101 I think of that as very much AI not this is some rule or something there's nothing rule-based a bit on that right it's just you're you're searching through a space and figure now are you optimizing through a space and figure out what seems to be the right thing to do and I think it's hard to just do that because you need to learn models of the world and I think it's hard to just do the learning part where you don't you know you don't bother with any of that because then you're saying well I could do imitation but then when I go off distribution I'm really screwed or you can say I can do reinforcement learning which adds a lot of robustness but then you have to do either reinforce my learning in the real world which sounds a little challenging or that trial and error you know or you have to do reinforce millennion simulation and then that means well guess why do you need to model things at least to a to model people model the world enough that you you know whatever policy you get of that is like actually fine to roll out in the world and do some additional learning there so do you think simulation by the way just the the the quick tangent has a role in the human robot interaction space like is it useful seems like humans everything we've been talking about are difficult to model and simulate do you think simulation has a role in this space I do I think so because you can take models and train with them ahead of time for instance you can but the model sorry to interrupt the models are sort of human constructed or learned I think they have to be a combination because if you get some human data and then you say this is hog this is gonna be my model of per the person what are for simulation and training or for just deployment time and that's what I'm planning with as my model of how people work regardless if you take some data um and you don't assume anything else and you just say okay this is this is some data that I've collected let me fit a policy to help people work based on that what does to happen is you collected some data in some distribution and then now you're your robot it computes a best response to that right is sort what should I do if this is how people work and easily goes off of distribution where that model that you've built of the human completely sucks because out of distribution you have no idea right there's if you think of all the possible policies and then you take only the ones that are consistent with the human data that you've observed that still needs a lot of put a lot of things could happen outside of that distribution where you're confident then you know what's going on by the way this should you have gotten used to this terminology out of a distribution within the system machine learning terminology because it kind of assumes so distribution is referring to the the data that you States that you encounter they've noticed so far at training time yeah but it kind of also implies that there's a nice like statistical model that represents that data so odd a distribution feels like I don't know it it uh it raises to me falafel questions of how we humans reason out of distribution reasonable things that are completely we haven't seen before and so and what we're talking about here is how do we reason about what other people do in you know situations where we haven't seen them and somehow we just magically navigate that right you know I can anticipate what will happen in situations that are even novel in many ways and I have a pretty good intuition for I always get it right but you know and I might be a little uncertain and so on I think it's it's this that if you just rely on data you know you you just too many possibilities or too many policies out there that fit the data and by the way it's not just state it's clearly kind of history of stake has to really be able to anticipate what the person will do it kind of depends on what they've been doing so far cuz that's the information you need to kind of at least implicitly sort of say oh this is the kind of person that this is this probably what they're trying to do so anyway it's like you're trying to map history States so actually there's many mapping and history meaning like the last yes word the last few minutes or the last few months who knows who knows how much you need right in terms of your state is really like the positions of everything or whatnot and velocities who knows how much you need and then and then there's this there's so many mappings and so now you're talking about how do you regularize that space what priors do you impose or what's the inductive bias so you know there's all very related things to think about it on basically water assumptions that we should be making such that these models actually generalize outside of the data that we've seen and now you're talking about well I don't know what can you assume maybe you can assume that people like actually have intentions and that's what drives their actions maybe that's you know the right thing to do when you haven't seen data very nearby that tells you otherwise I don't know it's a very open question do you think so that one of the dreams of artificial intelligence was to solve common sense reasoning whatever the heck that means do you think something like common sense reasoning has to be solved in part to be able to solve this dance of human interaction the driving space or human robot interaction in general you have to be able to reason about these kinds of common-sense concepts of physics of you know all the things we've been talking about humans I don't even know how to express them with words but the bay the basics of human behavior a fear of death so like to me it's really important to encode in some kind of sense maybe not maybe it's implicit but it feels that it's important to explicitly encode the fear of death that people don't want to die because it seems silly but like that that the game of chicken that involves with the pedestrian crossing the street is playing with the idea of mortality like we really don't want to dies that's just like a negative reward I don't know I it just feels like all these human concepts have to be encoded did you share that sense or is just a lot simpler that I'm making out to be I think it might be simpler and I'm the first thing who likes to complicate is I think where we simpler than that um because it turns out for instance if you if you say model people in the very I don't call it traditional why I don't know if it's fair to look at it as a traditional way but but you know calling people as okay they're irrational somehow the utilitarian perspective well in that once you say that they you automatically capture that they have an incentive to keep on being you know Stuart um likes to stay you can't fetch the coffee if you're dead Russell that's a good night so when when you're sort of cheating agents as having these objectives these incentives humans or artificial you're kind of implicitly modeling that they'd like to stick around so that they can accomplish those goals um so I think I think in a sense maybe that's what draws me so much to the rationality framework even though it's so broken we've been able to it's been such a useful perspective and like we were talking about earlier what's the alternative I give up and go home or you know I just use complete black boxes but then I don't know what to assume out of distribution that come back to this um it's just it's been a very fruitful way to think about the problem and a very more positive way right these people aren't just crazy maybe they make more sense than we think but um but I think we also have to somehow be ready for it to be to be wrong be able to detect when these assumptions aren't holding be all of that stuff let me ask sort of an another small side of this that we've been talking about the pure autonomous driving problem but there's also relatively successful systems already deployed out there in what you may call like level two autonomy or semi autonomous vehicles whether that's test autopilot of work quite a bit with Cadillac super guru system which has a driver facing camera that detects your state there's a bunch of basically Lane centering systems what's your sense about this kind of way of dealing with the human robot interaction problem by having a really dumb robot and and relying on the human to help the robot out to keep them both alive is that is that from the research perspective how difficult is that problem and from a practical deployment perspective is that a fruitful way to approach this human robot interaction problem I think what we have to be careful about there is to not me it seems like some of these systems not all are making this underlying assumption that if so I'm a driver and I'm now really not driving but supervising and my job is to intervene right and so we have to be careful with this assumption that when I'm if I'm supervising I will be just as safe as when I'm driving like that I will you know if I if I wouldn't get into some kind of accident if I'm driving I will be able to avoid that axis and when I'm supervising to and I think I'm concerned about this assumption from a few perspectives so from a technical perspective it's that when you'll add something kind of take control and do its thing and it depends on what that thing is obviously and how much is taking on how what things are you trusting it to do but if you let it do its thing and take control it will go to what we might call off policy from the person's perspective States so stays to the person wouldn't actually find themselves in if they were the ones driving and the assumption that the person functions just as well there as they function in the states that they would normally encounter is a little questionable now another part is the kind of the human factor side of this which is that I don't know about you but I think I definitely feel like I'm experiencing things very differently when I'm actively engaged in the task versus when I'm a passive observer even if I try to stay engaged right it's very different than when I'm actually actively making decisions and you see this in life in general like you see students who are actively trying to come up with the answer learn this thing better than when they're passively told the answer I think that's some more related and I think people have studied this in human factors for airplanes and I think it's actually fairly established that these two are not the same so I and that point because I've gotten a huge amount of heat on this and I stand by it okay because I know the human factors community well and the work here is really strong and there's many decades of work show exactly what you're saying nevertheless I've been continuously surprised that much of the predictions of that work has been wrong and what I've seen so what we have to do I still agree with everything you said but we have to be a little bit more open-minded so the the I'll tell you there's a few surprising things that super villi kever ething you said to the word is actually exactly correct but it doesn't say what you didn't say is that these systems are you said you can't assume a bunch of things but we don't know if he says are fundamentally unsafe that's still unknown if there's there's a lot of interesting things like I'm surprised by the fact not the fact that what seems to be anecdotally from well from large data collection that we've done but also from just talking to a lot of people when in the supervisory role of semi autonomous systems that are sufficiently dumb at least which is that might be a key element is the systems not to be dumb the people are actually more energized as observer so they're actually better they're they're better at observing the situation so there might be cases in systems if you get the interaction right or you as a supervisor will do a better job with the system together I agree I think that is actually really possible I guess mainly I'm pointing out that if you do it naively you're implicitly assuming something that assumption might actually really be wrong but I do think that if you explicitly think about what the agent reducers that the person still stays engaged what the so that you essentially empower the person do more than they could that's the really the goal right is you still have a driver so you want to empower them to be so much better than they would be by themselves and that's different it's a very different mindset then I want them to basically not join but be ready to sort of take over so one of the interesting things we'll be talking about is the rewards that they seem to be fundamental to the way robots behaves so broadly speaking we've been talking about utility function saw but you comment on how do we approach the design of reward functions like how do we come up with good reward function [Laughter] this is you know I used to think I think about how well it's actually really hard to specify rewards for interaction because and it's really supposed to be what the people want and then you really you know we talked about how you have to customize what you want to do to the end user but I kind of realized that even if you take the interactive component away it's still really hard to design reward functions so what do I mean by that I mean if we assumed this survey I paradigm in which there's an agent and his job is to optimize some objectives some reward utility lost whatever cost if you write it out maybe it's a sad depending on situation or whatever it is if you write it out and then you deploy the agent you'd want to make sure that whatever you specified incentivizes the behavior you want from the agent in any situation that the agent will be faced with right so I do motion planning on my robot arm I specify some cost function like you know this is how far away should try to stay so much amount of stay away from people and it so much it matters to be able to be efficient and blah blah blah Ryan I need to make sure that whatever I specified those constraints or trade-offs or whatever they are that when the robot goes and solves that problem in every new situation that behavior is the behavior that I want to see and what I've been finding is that we have no idea how to do that but basically what I can do is I can sample I can think of some situations that I think are representative of what the robot will face and I can turn and add and tune some reward function until the optimal behavior is what I want on those situations which first of all is super frustrating because you know through the miracle of AI we've taken we don't have to specify rules for behavior anymore right the saying before the robot comes up with the right thing to do you plug in this situation it optimizes writing that situation it optimizes but you have to spend still a lot of time and actually defining what it is that that criterion should be make sure you didn't forget about 50 bazillion things that are important and how they all should be combining together to tell the robot what's good and when it's bad and how good and how bad and so I think this is this is a lesson that I don't know kind of I guess I close my eyes to it for a while cuz I've been you know tuning cost functions for 10 years now but it it's it really strikes me that yeah we've moved the tuning and like designing of features or whatever from the behavior side into the reward side and yes I agree that there's way less of it but it still seems really hard to anticipate any possible situation and make sure you specify a reward function that when optimized will work well in every possible situation so so you're kind of referring to unintended consequences or just in general any kind of suboptimal behavior that emerges outside of the things you said about out of distribution suboptimal behavior that is you know actually optimal I mean this I guess the idea of unintended consequence you know it's I've don't respect what you specified but it's not what you want and there's a difference between those but that's not fundamentally a robotics problem it is a human problem so like that's the thing yeah right so there is this thing called good hearts law which is you start a metric for an organization and the moment it becomes on target that people actually optimize for it's no longer a good metric well what's it called the good hearts law good hearts Allah so the moment you specify a metric it stops doing his job yeah it stops doing his job um so there's yeah there's such a thing as off or optimizing for sayings and and you know failing to to think ahead of time of all the possible things that might be important and so that's so that's interesting because you story I work a lot on every word learning from the perspective of customizing to the end user but it really seems like it's not just the interaction with the end user that's a problem of the human and the robot collaborating so that the robot can do what the human one's right that's kind of back and forward the robot probing the person being informative all of that stuff might be actually just as applicable to this kind of maybe new form of human robot interaction which is the interaction between the robot and the expert programmer a roboticist designer in charge of actually specifying what the hectic wants should do a task for this professor that's so cool like collaborating on the reward right collaborating on the reward design and so what what does it mean right what is it when we think about the problem not as someone specifies all of your job is to optimize and we start thinking about your in this interaction and this collaboration and the first thing that comes up is when the person specifies a reward it's not you know gossip was not like the letter of the law it's not the definition of the reward function you should be optimizing because they're doing their best but they're not some magic perfect Oracle and the sooner we start understanding that I think the sooner we'll get tomorrow but instead of robots that function better in different situations and then then you have kind of say okay well it's it's almost like the robots are over learning over you're putting too much weight on the reward specified by definition and maybe leaving a lot of other information on the table like what are other things we could do to actually communicate to the robot about what we want them to do besides attempting to specify a reward phone yeah you have this awesome and again it looks the poetry of leaked information you mentioned humans leaked information about what they want you know leaked reward signal for the for the robot so how do we detect these leaks yeah what are these leaks are they just I don't know that those words recently saw it read it I don't know where from you and that's gonna stick with you for a while for some reason because it's not explicitly expressed it kind of leaks in directly from our behavior we do yeah absolutely so I think maybe something surprising bits right so we were talking to before about our my robot arm it needs to move around people carry stuff put stuff away all of that and now imagine that you know the robot has some initial objective that the programmer gave it so they can do all these things functional it's capable of doing that and now I noticed that it's doing something and maybe it's coming too close to me alright and maybe I'm the designer maybe I'm the end-user and this robot is now in my home and I push it away so I push away cuz you know it's a it's a reaction to what the robot is currently doing and this is what we call physical human robot interaction and now there's a lot of there's a lot of interesting work on how do you respond to physical human robot interaction why should the robot do if such an event occurs and there's sort of different schools of thought it's well you know you can sort of treat it to control theoretical and say this is a disturbance that you must reject you can sort of treat it more a kind of heuristic Leon sorry I'm gonna go into some like gravity compensation mode so that means very maneuverable around I'm gonna go in the direction that the person push me and and to us part of realization has been that that is signal that communicates about the reward because if my robot was moving in an optimal way and I intervened that means that I disagree which is notion of optimality whatever he thinks is optimal is not actually optimal and sort of optimization problems aside that means that the cost versus reward function is is incorrect or at least is not what I wanted to be how difficult a signal to to inter to make actionable so like I it cuz this connects to our Thomas vehicle discussion what they're in the semi autonomous vehicle or autonomous V go on a safety driver disengages the car like they could have disengaged it for a million reasons yeah yeah so that's true again it comes back to Kenya can you structure a little bit of your assumptions about how human behavior relates to what they want and you you know you can't one thing that we've done is literally just treated this external torque that they applied as you know when you take that and you add it with what the torque the robot was already applying that overall action is probably relatively optimal respect to whatever it is that the person wants and then that gives you information about what it is that they want so you can learn that people want you to stay further away from them now you're right that there might be many things that explain just at one signal that you might need much more data than that for the person be able to shape your reward function over time you can also do this info gathering stuff that we were talking about now now we've done that in that context just to clarify but it's definitely somebody thought about where you can have the robot start acting in a way like if there's a bunch of different explanations right it moves in a way where it sees if you corrected in some other way or not and then kind of actually plans its motion so that it can disambiguate then collect information about what you want anyway so that's one way that's cut a sort of leaked information maybe even more subtle leaked information is if I just press the e stop right I just I'm doing it out of panic because the robot is about to do something bad there's again information there right okay the robot should definitely stop but it should also figure out that whatever was about to do was not good and in fact it was so not good then stopping and remaining stop for a while was better a better trajectory for it than whatever it is that it was about to do and that again is information about what are my preference is what do I want speaking of East ops what are your expert opinions on the Three Laws of Robotics um Isaac Asimov don't harm humans obey orders protect yourself I mean it's a it's a such a silly notion but I speak to so many people these days just regular folks just I don't know my my parents and so on about robotics and they kind of operate in that space of you know imagining our future with robots and thinking what are the ethical how do we get that dance right I know the three laws might be a silly notion but do you do you think about like what Universal reward functions that might be that we should enforce on the robots of the future or is that a little too far out and it doesn't or is the mechanism that you just described you shouldn't be three laws it should be constantly adjusting kind of thing I think it should constantly be adjusting I think that you know the issue with the laws is I don't even you know they're words and I have to write math right and have to translate them into math what does it mean to us harm me what right because we just talked about how you try to say what you want but you don't always get it right and you want these machines to do what you want not necessarily exactly what your literacy you want them you don't want them to take you literally you want to take what you're saying and interpret it in context and that's what we do with the specified rewards we don't take them literally anymore from the designer we not we as a community we as you know some members are like we and in some of our collaborators like Peter bol and Stuart Russell we sort of say okay the designer specified this thing but I'm gonna interpret it not as this is universal reward function that I shall always optimize always and forever but as this is good evidence about what the person wants and I should interpret that evidence in the context of these situations that it was specified for because ultimately that's what the designers thought about that's what they had in mind and really them specifying a reward function that works for me in all these situations is really kind of telling me that whatever behavior that incentivizes must be good behavior respect to the thing that I should actually be optimizing for and so now the robot kinda has uncertainty about what it is that it should be what its reward function is and then there's all these additional signals we've been finding that it can kind of continually learn from and adapt its understanding of what people want every time the person corrected maybe they demonstrate maybe they stopped hopefully not right one really really crazy one is the environment itself like our world you don't it's not you know you observe our world and and the state of it and it's not that you're seeing behavior and you're saying how people are making decisions that are rational bla bla bla it's but but but our world is something that we've been acting when according to our preferences so I have this example where like the robot walks into my home and my shoes are laid down on the floor kind of in a line right it took effort to do that even though the robot doesn't see me doing this you know actually aligning the shoes it should still be able to figure out that I want the shoes online because there's no way for them to have magically instantiated themselves in that way someone must have actually just a good time to do that so it must be important so the environment actually tells the varlets information at least information I mean the environment is the way it is because humans some are manipulated is so you have to kind of reverse engineer the narrative that happened to create the environments it is and that leaks the yeah yeah yeah mission yeah you have to be careful yeah right because because people don't have the bandwidth to do everything so just because you know my house is messy doesn't mean that I want it to be messy right but that just shouldn't decide you know I didn't put the effort into that I put the effort into something else so the robot should figure out well that's something else was more important but it doesn't mean that you know the house being messy is not so it's a little subtle but yeah we really think of it the state itself is kind of like a choice that people implicitly made on how they want their world what book or books technical fiction or philosophical had when you like look back your life had a big impact maybe it was a turning point was inspiring maybe we're talking about some silly book that nobody in their right mind would want to read or maybe it's a book that you would recommend to others to read or maybe those could be two different recommendations that of books that could be useful for people on their journey when I was in it's kind of a personal story when I was in 12th grade I got my hands on a PDF copy in Romania of Russell Norvig a I modern approach I didn't know anything about AI at that point I was you know I had watched the movie The Matrix and and so I started going through this thing and you know you were asking in the beginning what are what are you and just it's it you know it's math and it's algorithms what's interesting it was so captivating this notion that you could just have a goal and figure out your way through a kind of a messy complicated situation so what sequence of decisions you should make art autonomously to achieve that goal that was so cool I'm you know I'm biased but that's a cool book yeah you can convert you know the goal the goal of and tell it the process of intelligence and mechanize it I had the same experience I was really interested in psychiatry and trying to understand human behavior and then AI a modern approach is like wait you can just reduce it all yeah so that's and I think that's stuck with me cuz you know a lot of what I do a lot of what we do in my lab is write math about human behavior combine it with data and learning put it all together give it to robots to plan wit and you know hope that instead of writing rules for the robots writing heuristics designing behavior they can actually autonomously come up with the right thing to do around people that's kind of our you know that's our signature move it's we wrote some mass and then instead of kind of hand crafting this and that and that and the robots figuring stuff out and isn't that cool and I think that is the same enthusiasm that I got from there I figured out how to reach that goal in that graph isn't that cool so apologize for the romanticized questions but and the silly ones if a doctor gave you five years to live sort of emphasizing the finiteness of our existence what would you try to accomplish it's like my biggest nightmare by the way I really like living I really don't like dying of being told that I'm gonna die sorry Dylan got enough for a second do you I mean do you meditate or ponder on your mortality on our human the fact that this thing ends it seems to be a fundamental feature do you think of it as a feature or a bug - is it you you said you don't like the idea of dying but if I were to give you a choice of living forever like you're not allowed to die yeah now I'll say that I'm wandering forever but I watch this show it's very still it's called a good place and they reflect a lot on this and you know the the moral of story is that you have to make the afterlife be finite - because otherwise people just like wall-e so so I think the finance helps but but yeah it's just um you know I don't I don't I'm not a religious person I don't think that there's something after and so I think it just ends and you stop existing and I really like existing it's just it's such a great privilege to exist that that yeah it's just I think that's very part I still think that we we like existing so much because it ends mm-hmm and that's so sad like it's so sad to me every time I got find almost everything about this life beautiful like the silliest most mundane things are just beautiful and I think I'm cognizant of the fact that I find it beautiful because it ends like it and it's so I don't know I don't know how to feel about that I also feel like there's a lesson in there for robotics an AI that is not like the finite of things seems to be a fundamental nature of human existence I think some people sort of accuse me of just being Russian and melancholic and romantic or something but that seems to be a fundamental nature of our existence that should be incorporated in our reward functions but anyway if you were speaking of reward functions if you only had five years what would you try to accomplish this is the thing I I'm thinking about this question and have a pretty joyous moment because I don't know that i would change mine listen I'm what I'm I'm you know I'm trying to make some contributions to how we understand human AI interaction I don't think I would change that um maybe I'll check you know I take more trips to the Caribbean or something but I try to spend time so yeah I mean I try to to do the things that bring me joy and thinking about these things bring me joy is d'amérique Ando think you know don't do stuff that doesn't spark joy for the most part I do things that spark joy maybe I'll do like less service in the department or something but but no I mean I think I have amazing colleagues and amazing students and amazing family and friends and kind of spending time and some balance with all of them is what I do and I that's what I'm doing already so I don't know that I would really change anything so on the spirit of positiveness oh what small act of kindness if one pops to mind where you one's shown you will never forget mmm when I was in high school my friends my my classmates did some tutoring we were gearing up for our baccalaureate exam and we they did some tutoring on well someone math someone whatever I was comfortable enough with with some of those subjects but physics was something that I hadn't focused in a while and so they were all working with this one teacher and I started working with that teacher her name is Nicole McConnell and she she was the one who kind of opened up this whole world for me because she sort of told me that I should take the SATs and apply to go to college abroad and you know do better on my English and all of that and when it came to well financially I couldn't my parents couldn't really afford to do all these things she started tutoring me on physics for free and on top of that sitting down with me to kind of train me for SATs and all that jazz that she had experience with Wow and obviously that has taken you to be to here today also to one of the world's experts and robotics it's funny those little yeah dude use these small word for no reason really kindness just out of karma wanting to support someone yeah yeah so we talked a ton of our reward functions let me talk about the the most ridiculous big question what is the meaning of life what's the reward function under which we humans operate like what may be to your life may be broader to human life in general what do you think what gives life fulfillment purpose happiness meaning you can't even ask that question with a straight face that's how ridiculous I can't like him okay so you know you're gonna try to answer it anyway aren't you so I was in a planetarium once yes and you know they show you the thing and these do man is zoom out and this whole like you're a speck of dust kind of thing I think that was conceptualizing that we're kind of you know what our humans were just on this little planet whatever we don't matter much in the grand scheme of things and then my mind got really blown cuz this doctor they doctored this multi-verse this theory where they kind of zoomed out and were like this is our universe and then like there's a bazillion other ones and it stays pop in and out of existence so like our whole thing that's that we can't even fathom how big it is was like a blimp that went in and out and I thought I was like okay clearly what we should be doing is try to impact whatever local thing we can impact our communities leave a little bit behind they're our friends our family our local communities and just try to be there for other humans cuz I just everything beyond that seems ridiculous I mean are you like how do you make sense of these multiverses like are you inspired by the immensity of it do you you can it is there like is it amazing to you or is it almost paralyzing in this in the mystery of it it's frustrating I'm frustrated by my inability to comprehend it feels very frustrating it's look there's there's some stuff that you know we should time blah blah blah that we should really be understanding and I definitely don't understand it but you know the the amazing physicists of the world have a much better understanding than me Don and the grand scheme of things so it's very frustrating it's just it feels like our brain don't have some fundamental capacity yeah well yet or ever I don't know but well this one of the dreams of artificial intelligence is to create systems that will aid expand our cognitive capacity in order to understand the build the theory of everything when the physics and understand what the heck these multiverses are so I think there's no better way to end it than talking about the meaning of life and the fundamental nature of the universe and akka is a huge honor one of the my favorite conversations I've had I really really appreciate your time thank you for talking to thank you for coming come back again thanks for listening to this conversation with anchor dragon and thank you to our presenting sponsor cash app please consider supporting the podcast by downloading cash app and using code lex podcast if you enjoy this podcast subscribe on youtube review it with five stars on a podcast supported on patreon or simply connect with me on Twitter and lex friedman and now let me leave you with some words from Isaac Asimov your assumptions are your windows in the world scrub them off every once in a while or the light won't come in thank you for listening and hope to see you next time you