Peter Norvig: Artificial Intelligence: A Modern Approach | Lex Fridman Podcast #42
the following is a conversation with peter norvig he's a director of research at google and the co-author with stuart russell of the book artificial intelligence and modern approach that educated and inspired a whole generation of researchers including myself to get into the field of artificial intelligence this is the artificial intelligence podcast if you enjoy it subscribe on youtube give it five stars on itunes support on patreon or simply connect with me on twitter at lex friedman spelled f-r-i-d-m-a-n and now here's my conversation with peter norvig most researchers in the ai community including myself own all three editions red green and blue of the uh artificial intelligence a modern approach it's a field-defining textbook as many people are aware that you wrote with stuart russell how has the book changed and how have you changed in relation to it from the first edition to the second to the third and now fourth edition as you work on it yeah so it's been a lot of years a lot of changes one of the things changing from the first to maybe the second or third was just the rise of uh computing power right so i think in the in the first edition we said uh here's predicate logic but uh that only goes so far because pretty soon you have millions of uh short little predicate expressions and they couldn't possibly fit in memory uh so we're going to use first order logic that's more concise and then we quickly realized oh predicate logic is pretty nice because there are really fast sat solvers and other things and look there's only millions of expressions and that fits easily into memory or maybe even billions fit into memory now so that was a change of the type of technology we needed just because the hardware expanded even to the second edition resource constraints were loosened significantly yeah yeah and that was the early 2000s second edition right so 95 was the first and then 2000 2001 or so and then moving on from there i think we're starting to see that again with the gpus and then more specific type of machinery like the tpus and using custom asics and so on for deep learning so we're seeing another advance in terms of the hardware then i think another thing that we especially notice this time around is in all three of the first editions we kind of said well we're going to find ai as maximizing expected utility and you tell me your utility function and now we've got 27 chapters worth of cool techniques for how to optimize that i think in this edition we're saying more you know what maybe that optimization part is the easy part and the hard part is deciding what is my utility function what do i want and if i'm a collection of agents or a society what do we want as a whole so you touch that topic in this edition you get a little bit more into utility yeah that's really interesting on a technical level we're almost pushing the philosophical i guess it is philosophical right so we we've always had a philosophy chapter which which i was uh glad to that we were supporting and now it's less kind of the uh you know chinese room type argument and more of these uh ethical and societal type issues so we get into uh the issues of fairness and bias and uh and just the issue of aggregating utilities so how do you encode human values into a utility function is is this something that you can do purely through data in a learned way or is there some systematic obviously there's no good answers yet there's just uh beginnings to this uh to even opening doors so there is no one answer yes there are techniques uh to try to learn that so we talk about inverse reinforcement learning right so reinforcement learning uh you take some actions you get some rewards and you figure out what actions you should take in inverse reinforcement learning you observe somebody taking actions and you figure out uh well that this must be what they were trying to do if they did this action it must be because they want it of course there's restrictions to that right so lots of people take actions that are self-destructive uh where they're they're suboptimal in certain ways so you don't want to learn that right you want to uh somehow learn the uh the perfect actions uh rather than the ones they actually take so so that's a challenge uh for that field then another big part of it is just kind of uh theoretical of saying uh what can we accomplish and so you look at like this this work on the uh programs to uh predict recidivism and decide uh you know who should get parole or who should get bail or whatever uh and how are you gonna evaluate that and one of the big issues is fairness across protected classes protected classes being things like uh sex and race and so on and uh so two things you want is you want to say well if i get a score of say uh six out of ten then i want that to mean the same whether no matter what race i'm on yes right so i want to have a 60 percent chance of reoccurring uh regardless uh and the makers of the one of the makers of a commercial program to do that says that's what we're trying to optimize and look we achieved that we've we've reached that kind of balance and then on the other side you also want to say well if if it makes mistakes i want that to affect both sides of the protected class equally and it turns out they don't do that right so they're they're twice as likely to make a mistake that would harm a black person over a white person so that seems unfair so you'd like to say well i want to achieve both those goals and then it turns out you do the analysis and it's theoretically impossible to achieve both those goals so you have to trade them off one against the other so that analysis is really helpful to know what you can aim for and how much you can get that you can't have everything but the analysis certainly can't tell you where should we make that trade-off point but nevertheless then we can uh as humans deliberate where that trade-off should be yeah so at least we now we're we're arguing an informed way we're not asking for something impossible we're saying uh here's where we are and and here's what we aim for and this strategy is better than that strategy so that's i would argue is a really powerful and really important first step but it's a doable one sort of removing uh undesirable degrees of bias in uh in systems in terms of protected classes and then there's something i listen to your uh commencement speech or there's some fuzzier things like you mentioned angry birds yeah do you want do you want to create systems that feed the dopamine enjoyment that feed that optimize for you returning to the system enjoying the moment of playing the game of getting likes or whatever this kind of thing or some kind of long-term improvement right is if are you even thinking about that that's ex that's really going to the philosophical area i think that's a really important issue too certainly thinking about that i i don't think about that as a as an ai issue as much but as you say you know the point is we've built this society in this infrastructure where we say we have a marketplace for attention and uh we've decided as a society that we like things that are free and so we want all uh apps on our phone to be free uh and that means they're all competing for your attention and then eventually they they make some money some way through uh ads or in-game sales or whatever but they can only win by defeating all the other apps by instilling your attention and we build a marketplace where it seems like they're working against you rather than working with you and i'd like to find a way where we can change the playing field so we feel more like well these things are on my side yes they're letting me have some fun in the short term but they're also helping me in the long term rather than competing against me and those aren't necessarily conflicting objectives they're just the incentives the direct current incentives as we try to figure out this whole new world seem to be on uh the easier part of that which is feeding the dopamine the rush right but uh let me take a quick step back at the beginning of the artificial intelligence and modern approach book of writing so here you are in the 90s when you first sat down with stuart to write the book to cover an entire field which is one of the only books that successfully done that for ai and actually in a lot of other computer science fields you know it's a dif it's a it's a huge undertaking so it must have been quite daunting what was that process like did you envision that you would be trying to cover the entire field was there a systematic approach to it that was more step by step how did it feel so i guess it came about you know go to lunch with the other ai faculty at berkeley and we'd say uh you know the field is changing seems like the current books are a little bit behind nobody's come out with a new book recently we should do that and everybody said yeah yeah that's a great thing to do and we never did anything right and then i ended up heading off to uh industry i went to uh sun labs so i thought well that's the end of my possible academic publishing career but i met stuart again at a conference like a year later and said you know that book we were always talking about you guys must be half done with it by now right he said well we keep talking we never do anything so i said well you know we should do it and i think the reason is that we all felt it was a time where the field was changing and that was in two ways so you know the good old-fashioned ai was based uh primarily on boolean logic you had a few tricks to deal with uncertainty and it was based primarily on knowledge engineering then the way you got something done is you went out you interviewed an expert and you wrote down by hand everything they knew and we saw in in 95 that the field was changing in in two ways one we're moving more towards probability rather than boolean logic and we're moving more towards machine learning rather than knowledge engineering uh and the other books uh hadn't caught that way if they were still in the uh more in the in the old school although so certainly they had part of that on the way but we said if we start now completely taking that point of view we can have a different kind of book and we were able to put that together and uh what was literally the process if you remember did you start writing a chapter did you outline yeah i guess i guess we did an outline and then we sort of assigned chapters to each person at the time uh i had moved to boston and stewart was in berkeley so basically uh we did it uh uh over the internet and uh you know that wasn't the same as doing it today it meant you know dial-up lines and telnetting in and you know you you telnetted into one shell and you type cat file name and you hoped it was captured at the other end and certainly you're not sending uh images and figures back and forth right right that didn't work but you know did you anticipate where the field would go from that day from from the 90s did you see the growth into learning-based methods into data-driven methods that followed in the future decades we certainly thought that learning was important i guess we we missed it as uh being as important as it as it is today we missed this idea of big data we missed that uh uh the idea of deep learning hadn't been invented yet we could have uh taken the book from a complete uh machine learning point of view right from the start we chose to do it more from a point of view of we're going to first develop different types of representations and we're going to talk about different types of environments of is it fully observable or partially observable and is it deterministic or stochastic and so on and we made it more complex along those axes rather than focusing on the machine learning axis first do you think you know there's some sense in which the deep learning craze is extremely successful for a particular set of problems and you know eventually it's going to in the general case hit challenges so in terms of the difference between perception systems and robots that have to act in the world do you think uh we're going to return to ai modern approach type breadth in addition five and six yeah in uh in future decades do you think deep learning will take its place as a chapter and as in his bigger uh view of ai yeah i think we don't know yet how it's all going to play out so uh in the new edition uh we have a chapter on deep learning uh we got ian goodfellow to be the guest author for that chapter so he said he could condense his whole deep learning book into one chapter i think he did a great job we were also encouraged that he's you know we gave him the old neural net chapter and said have fun with it modernize that and he said you know half of that was okay that certainly there's lots of new things that have been developed but some of the core was still the same so i think we'll gain a better understanding of what you can do there i think we'll need to incorporate all the things we can do with the other technologies right so deep learning started out convolutional networks and very close to perception uh and has since moved to be uh to be able to do more with actions and some degree of longer term planning but we need to do a better job with representation than reasoning and one-shot learning and so on and well i think we don't know yet how that's going to play out so do you think looking at the some success but certainly uh eventual demise the partial demise of experts to symbolic systems in the 80s do you think there is kernels of wisdom in the work that was done there with logic and reasoning and so on that will rise again in your view so certainly i think the idea of representation and reasoning is crucial that you know sometimes you just don't have enough data about the world to learn de novo so you've got to have some idea of representation whether that was programmed in or told or whatever and then be able to take uh steps of reasoning i i think the problem uh with the you know the good old-fashioned ai was uh one we tried to base everything on these uh symbols that were atomic and that's great if you're like trying to define the properties of a triangle right because they have necessary insufficient conditions uh but things in the real world don't the real world is is messy and doesn't have sharp edges and atomic symbols do so that was a poor match and then the other aspect was that the reasoning was universal and applied anywhere which in some sense is good but it also means there's no guidance as to where to apply it and so you you know you started getting these paradoxes like uh well if i have a mountain and i remove one grain of sand then it's still a mountain and but if i do that repeatedly at some point it's not right and with logic you know there's nothing to stop you from applying things uh repeatedly but maybe with something like deep learning and i don't really know what the right name for it is we could separate out those ideas so one we could say uh you know a mountain isn't just an atomic notion it it's some sort of something like uh word embedding that uh uh has a a more complex representation yeah and secondly we could somehow learn yeah there's this rule that you can remove one grain of sand and you can do that a bunch of times but you can't do it a near infinite amount of times but on the other hand when you're doing induction on the integer sure then it's fine to do it an infinite number of times and if we could uh somehow we have to learn when these strategies are applicable rather than having the strategies be completely neutral and available everywhere anytime you use neural networks anytime you learn from data or form representation from data in an automated way it's not very explainable as to or it's not introspective to us humans in terms of uh how this neural network sees the world where why does it succeed so brilliantly on so many in so many cases and fail so miserably in surprising ways and small so what do you think is this is uh the future there can simply more data better data more organized data solve that problem or is there elements of symbolic systems that need to be brought in which are a little bit more explainable yeah so i prefer to talk about trust and uh validation and verification rather than just about explainability and then i think uh explanations are one tool that you use towards those goals and i think it is important issue that we don't want to use these systems unless we trust them and we want to understand where they work and where they don't work and in an explanation can be part of that right so i apply for loan and i get denied i want some explanation of why and you have in europe we have the gdpr that says you're required to be able to get that but on the other hand explanation alone is not enough right so you know we're used to dealing with people and with the organizations and corporations and so on and they can give you an explanation then you have no guarantee that that explanation relates to reality right right so the bank can tell me well you didn't get the loan because you didn't have enough collateral and that may be true or it may be true that they just didn't like my religion or or something else i can't tell from the explanation and that's that's true whether the decision was made by computer or by a person so i want more i do want to have the explanations and i want to be able to uh have a conversation to go back and forth and said well you gave this explanation but what about this and what would have happened if this had happened and uh what would i need to change that so i think a conversation is is a better way to think about it than just an explanation as a single output and i think we need testing of various kinds right so in order to know was the decision really based on my collateral or was it based on my uh religion or skin color or whatever i can't tell if i'm only looking at my case but if i look across all the cases then i can detect a pattern all right right so you want to have that kind of capability uh you want to have these adversarial testing right so we thought we were doing pretty good at object recognition in images we said look we're at sort of pretty close to human level performance on imagenet and so on and then you start seeing these adversarial images and you say wait a minute that part is nothing like human performance okay you can mess with it really easily you can mess with it really easily right and uh yeah you could do that to humans too right so in a different way perhaps right humans don't know what color the dress was right and so they're vulnerable to certain attacks that are different than the attacks on the on the machines but the you know the tax on the machines are so striking uh they really change the way you think about what we've done right and the the way i think about it is i think part of the problem is we're seduced by uh our low dimensional metaphors right yeah so you know you look like that phrase you look in uh in a textbook and you say okay now we've mapped out the space and you know uh cat is here and dog is here and maybe there's a tiny little spot in the middle where you can't tell the difference but mostly we've got it all covered and if you believe that metaphor uh then you say well we're nearly there and uh you know there's only going to be a couple adversarial images but i think that's the wrong metaphor and what you should really say is it's not a 2d flat space that we've got mostly covered it's a million dimension space and a cat is this string that goes out in this crazy bath and if you step a little bit off the path in any direction you're in nowhere's land and you don't know what's going to happen and so i think that's where we are and now we've got to deal with that so uh it wasn't so much an explanation but it was an understanding of what the models are and what they're doing and now we can start exploring how do you fix that yeah validating the robustness of the system so on but take it back to the this uh this word trust uh do you think we're a little too hard on our robots in terms of uh the standards we apply so you know of uh there's a dance there's a there's a there's a dance and nonverbal and verbal communication between humans you know if we apply the same kind of standard in terms of humans you know we trust each other pretty quickly uh you know you and i haven't met before and there's some degree of trust right that nothing's gonna go crazy wrong and yet to ai when we look at ai systems where we seem to approach uh through skepticism always always and it's like they have to prove through a lot of hard work that they're even worthy of uh even inkling of our trust do it what do you what do you think about that how how do we break that barrier close that gap i think that's right i think that's a big issue uh just listening uh my friend uh mark moffat is a naturalist and he says uh the most amazing thing about humans is that you can walk into a coffee shop or a a busy street in a city and there's lots of people around you that you've never met before and you don't kill each other yeah he says chimpanzees cannot do that yeah right right if the chimpanzee's in a situation where here's some that aren't from my tribe bad things happen especially in a coffee shop there's delicious food around you know yeah yeah but but we humans have figured that out yeah right uh and you know for the most part for the most part we still go to war we still do terrible things uh but for the most part we've learned to trust each other and live together uh so that's going to be important for our uh our ai systems as well and i th also i think uh you know a lot of the emphasis is on ai but in many cases ai is part of the technology but isn't really the main thing so a lot of what we've seen is more due to communications technology than ai ai technology yeah you want to make these good decisions but the reason we're able to have any kind of system at all is we've got the communication so that we're collecting the data and so that we can reach lots of people around the world i think that's a bigger change that we're dealing with speaking of reaching a lot of people around the world on the side of education you've uh one of the many things in terms of education you've done you taught the intro to artificial intelligence course that signed up 100 160 000 students is one of the first successful examples and massive of a mooc massive open online course what did you learn from that experience what do you think is the future of moocs of education online yeah it was a great fun doing it particularly uh being right at the start just because it was exciting and new but it also meant that we had less competition right so uh one of the things you hear about uh well the problem with moocs is uh the completion rates are are so low so there must be a failure and and i gotta admit i'm a prime contributor right i've probably started 50 different courses that i haven't finished but i got exactly what i wanted out of them because i had never intended to finish them i just wanted to dabble in a little bit either to see the topic matter or just to see the pedagogy of how are they doing this class so i guess the main thing i learned is when i came in i thought the challenge was information saying if i'm just take the stuff i want you to know and i'm very clear and explain it well then my job is done and good things are going to happen and then in doing the course i learned well yeah you got to have the information but really the motivation is the most important thing that if students don't stick with it it doesn't matter how good the content is and i think being one of the first classes we were helped by uh sort of exterior motivation so we tried to do a good job of making it enticing and setting up ways for uh you know the community to work with each other to make it more motivating but really a lot of it was hey this is a new thing and i'm really excited to be part of a new thing and so the students brought their own motivation and so i think this is great because there's lots of people around the world who have never had this before you know it would never have the opportunity to go to stanford and take a class or go to mit or go to one of the other schools but now we can bring that to them and if they bring their own motivation they can be successful in a way they couldn't before but that's really just the top tier of people that are ready to do that the rest of the people just don't see or don't have their motivation and don't see how if they push through and were able to do it what advantage that would get them so i think we've got a long way to go before we're able to do that and i think it'll be some of it is based on technology but more of it's based on the idea of community that you got to actually get people together some of the getting together can be done online i think some of it really has to be done in person to be able in order to build that type of community and trust you know there's an intentional mechanism that we've developed uh a short attention span especially younger people uh because sort of shorter and shorter videos online uh there's a whatever the the way the brain is developing now with people that have grown up with the internet they have a quite a short attention span so and i would say i had the same when i was growing up too probably for different reasons so i probably wouldn't have learned as much as i have if i wasn't forced to sit in a physical classroom sort of bored sometimes falling asleep but sort of forcing myself through that process to sometimes extremely difficult computer science courses what's the difference in your view between in-person education experience which you first of all yourself had and you yourself taught and online education and how do we close that gap if it's even possible yeah so i think there's two issues one is whether it's in person or online so it's sort of the physical location and then the other is kind of the affiliation right so you stuck with it in part because you were in the classroom and you saw everybody else was suffering right the same way you were but also because you were enrolled you had paid tuition sort of everybody was expecting you to stick with it society parents yeah peers right and so those are two separate things i mean you could certainly imagine i pay a huge amount of tuition and everybody signed up and says yes you're doing this uh but then i'm in my room and my classmates are in are in different rooms right we could have things set up that way so it's not just the online versus offline i think what's more important is the commitment that you've made and certainly it is important to have that kind of informal you know i meet people outside of class we talk together because we're all in it together i think that's uh really important both in keeping your motivation and also that's where some of the most important learning goes on so you want to have that maybe you know especially now we start getting into higher bandwidths and augmented reality and virtual reality you might be able to get that without being in the same physical place do you think it's possible we'll see a course at stanford for example that for students enrolled students is only online in the near future who are literally sort of that's part of the curriculum and there is no yeah so you're starting to see that i know uh georgia tech has a master's that's done that way oftentimes it's sort of they're creeping in in terms of a master's program or sort of um further education considering the constraints of students and so on but i mean literally is it possible that we just you know stanford mit berkeley all these places go online only in uh in the next few decades yeah probably not because you know they've got a big commitment to a physical campus sure right there's a momentum that's both financial and culturally right and and then there are certain things that just hard to do uh virtually right so you know we're in a field uh where uh if you have your own computer and your own paper and so on uh you can do the work anywhere uh but if you're in a biology lab or something uh you know you don't have all the right stuff at home right so our field programming you've also done a lot of you've done a lot of programming yourself in 2001 you wrote a great article about programming called teach yourself programming in 10 years sort of response to all the books that say teach yourself programming in 21 days so if you're giving advice to someone getting into programming today this is a few years since you've written that article what's the best way to undertake that journey i think there's lots of different ways and i think programming means more things now and i guess you know when i wrote that article i was thinking more about becoming a professional software engineer and i thought that's a you know a sort of a career-long field of study but i think there's lots of things now that people can do where programming is a part of solving what they want to solve without achieving that professional level status right so i'm not going to be going and writing a million lines of code but you know i'm a biologist or a physicist or something or even a historian and i've got some data and i want to ask a question of that data and i think for that you don't need 10 years right so there are many shortcuts to being able to answer those kinds of questions and and you know you see today a lot of emphasis on learning to code teaching kids how to code uh i think that's great uh but i wish they would change the message a little bit right so i think code isn't the main thing i don't really care if you know the syntax of javascript or if you can connect these blocks together in this visual language but what i do care about is that you can analyze a problem you can think of a solution you can carry out you know make a model run that model test the model see the results verify that they're reasonable ask questions and answer them right so it's more modeling and problem solving and you use coding in order to do that but it's not just learning coding for its own sake that's really interesting so it's actually almost in many cases it's learning to work with data to extract something useful out of data so when you say problem solving you really mean taking some kind of maybe collecting some kind of data set cleaning it up and saying something interesting about it which is useful in all kinds of domains and you know and i see myself being stuck sometimes in kind of the the old ways right so you know be working on a project maybe with a younger employee and we say oh well here's this new package that could help solve this problem and i'll go and i'll start reading the manuals and you know i'll be two hours into reading the manuals and then my colleague comes back and says i'm done yeah you know i downloaded the package i installed it i tried calling some things the first one didn't work the second one work now i'm done yeah and i say but i have 100 questions about how does this work and how does that work and they say who cares right i don't need to understand the whole thing i unders i answered my question it's a big complicated package i don't understand the rest of it but i got the right answer and i'm just it's hard for me to get into that mindset i want to understand the whole thing and you know if they wrote a manual i should probably read it and but that's not necessarily the right way i think i have to get used to dealing with more being more comfortable with uncertainty and not knowing everything yeah so i struggle with the same instead of the the spectrum between donald and don knuth yeah it was kind of the very you know before he can say anything about a problem he really has to get down to the machine code assembly yeah versus exactly what you said i've have several students in my group that uh you know 20 years old and they can solve almost any problem within a few hours that would take me probably weeks because i would try to as you said read the manual so do you think the nature of mastery you're you're mentioning biology sort of outside disciplines applying programming but computer scientists so over time there's higher and higher levels of abstraction available now so with uh this week there's a the tensorflow summit right so if you're if you're not particularly into deep learning but you're still a computer scientist uh you can accomplish an incredible amount with uh tensorflow without really knowing any fundamental internals of machine learning do you think the nature of mastery is is changing even for computer scientists like what it means to be an expert programmer yeah i think that's true you know we never really should have focused on programmer right because it's still it's the skill and what we really want to focus on is the result so we we built this uh ecosystem where the way you can get stuff done is by programming it yourself at least when i started that you know library functions meant you had square root and that was about it right everything else you built from scratch and then we built up an ecosystem where a lot of times well you can download a lot of stuff that does a big part of what you need and so now it's more a question of assembly rather than [Music] manufacturing and that's a different way of looking at problems from another perspective in terms of mastery and looking at programmers or people that reason about problems in a computational way so google is you know the from the hiring perspective from the perspective of hiring or building a team of programmers how do you determine if someone's a good programmer or if somebody again yeah i want to deviate from i want to move away from the word programmer but somebody who can solve problems of large-scale data and so on what's what's uh how do you build a team like that through the interviewing process yeah and i and i think uh as a company grows uh you get more uh expansive in the types of people you're looking for right so i think you know in the early days we'd interview people and the question we were trying to ask is uh how close are they to jeff dean and most people were pretty far away but we take the ones that were you know not that far away and so we got kind of a homogeneous group of people who are really great programmers then as a company grows you say well we don't want everybody to be the same to have the same skill set and so now we're uh hiring uh biologists in our health areas and we're hiring physicists we're hiring mechanical engineers we're hiring uh you know social scientists and ethnographers and people with different backgrounds who bring different skills so you have mentioned that you still may partake in code reviews given that you have a wealth of experience as you've also mentioned uh what errors do you often see and tend to highlight in the code of junior developers of people coming up now uh given your background from blisp to a couple decades of programming yeah that's a great question you know sometimes i try to look at the flexibility of the design of yes you know this api solves this problem but uh where is it going to go in the future who else is going to want to call this and uh you know are you making it easier for them to do that it's a matter of design is it documentation is it is it uh sort of an amorphous thing you can't really put it it's just how it feels if you put yourself in the shoes of a developer would you use this kind of thing i think it is how you feel right and so yeah documentation is good uh but it's but it's more a design question right if you get the design right then people will figure it out whether the documentation is good or not and and if the design is wrong then it'll be harder to use how have uh you yourself changed as a programmer over the years as in in a way we already started to say sort of you want to read the manual you want to understand the core of the syntax to the how the language is supposed to be used and so on but what's the evolution been like from the 80s 90s to today i guess one thing is you don't have to worry about the small details of efficiency as much as you used to right so like i remember uh i did my list book in the 90s and one of the things i wanted to do was say uh here's how you do an object system and uh basically we're going to make it so each object is a hash table and you look up the methods and here's how it works and then i said of course the real common lisp object system is much more complicated it's got all these efficiency type issues and this is just a toy nobody would do this in real life and it turns out python pretty much did exactly what i said yeah and said uh objects are just dictionaries and yeah they have a few little uh tricks as well but mostly you know the thing that would have been 100 times too slow in the 80s is now plenty fast for most everything so you had to as a programmer let go of perhaps an obsession that i remember coming up with of trying to write efficient code yeah that to say you know what really matters is the total time it takes to get the project done and most of that's going to be the programmer time so if you're a little bit less efficient but it makes it easier to understand and modify then that's the right trade-off so you've written quite a bit about lisp your book on programming is in lisp you you have a lot of code out there that's in lisp so myself and people who don't know what lisp is should look it up it's my favorite language for many ai researchers it is a favorite language the favorite language they never use these days so what part of the list do you find most beautiful and powerful so i think the beautiful part is the simplicity that in half a page you can define the whole language and other languages don't have that so you feel like you can hold everything in your head and then you know a lot of people say well then that's too simple you know here's all these things i want to do and you know my java or python or whatever has 100 or 200 or 300 different syntax rules and don't i need all those and lisp's answer was no we're only going to give you eight or so syntax rules but we're going to allow you to define your own and so that was a very powerful idea and i think this idea of saying i can start with my problem and with my data and then i can build the language i want uh for that problem and for that data and then i can make lists define that language so you uh you're sort of uh mixing levels and saying i'm simultaneously a programmer in a language and a language designer and that allows a better match between your problem and your eventual code and i think lis had done that better than other languages yeah it's a very elegant implementation of functional programming but why do you think lisp has not had the mass adoption and success of languages like python is it the parentheses is it all the parentheses yeah so i think a couple of things so one was i think it was designed for a single programmer or a small team and a skilled programmer who had the good taste to say well i'm i am doing language design and i have to make good choices and if you make good choices that's great if you make bad choices you can hurt yourself and it can be hard for other people on the team to understand it so i think there was a limit to the scale of the size of a project in terms of number of people that lisp was good for and as an industry we kind of grew beyond that i think it is in part the parentheses you know one of the jokes is the acronym for lisp is lots of irritating silly parentheses my acronym was lisp is syntactically pure saying all you need is parentheses and atoms but i remember you know so we had the the ai textbook and uh because we did it in the 90s we had we had pseudocode in the book but then we said well we'll have lisp online because that's the language of ai at the time and i remember some of the students complaining because they hadn't had lists before and they didn't quite understand what was going on and i remember one student complained i don't understand how this pseudocode corresponds to this lisp and there was a one-to-one correspondence between the the symbols in the code in the pseudocode and the only thing difference was the parentheses so i said it must be that for some people a certain number of left parentheses shuts off their brain yeah it's very it's very possible in that sense then python just goes the other way and so so that was the point at which i said okay can't have only lisp that's a language because i you know i don't want to you know you only got 10 or 12 or 15 weeks or whatever it is to teach ai and i don't want to waste two weeks of that teaching lisp so i say i got to have another language java was the most popular language at the time i started doing that and then i said it's really hard to have a one-to-one correspondence between the pseudocode and the java because java's so verbose so then i said i'm going to do a survey and find the language that's most like my pseudocode and turned out python basically was my pseudo code somehow i had channeled uh guido and designed a pseudocode that was the same as python although i hadn't heard of python uh at that point and from then on uh that's what i've been using because it's been a good match so what's the story in python behind pietudes your github repository with puzzles and exercises and python is pretty fun yeah just it seems like fun uh you know you know i like uh doing puzzles and i like uh being an educator i did a class with udacity uh udacity uh 212 i think it was it was basically problem solving uh using python and looking at different problems does pie tubes feed that class in terms of the exercises i was wondering what that yeah so the class the class came first yeah some of the stuff that's in pi tubes was write-ups of what was in the class and then some of it was just continuing to uh to work on new problems so what's the organizing madness of pi tubes is it just the collect a collection of cool exercises just whatever i thought was fun okay awesome so you were the director of search quality of google from 2001 to 2005. in the early days uh when there's just a few employees and when the company was growing like crazy right so i mean a google revolution has the way we discover share and aggregate knowledge so just this is uh this is one of the fundamental aspects of civilization right is information being shared and there's different mechanisms throughout history but google is just 10x improved that right and you're a part of that right people discovering that information so what were some of the challenges on the philosophical or the technical level in those early days it definitely was an exciting time and as you say we were doubling in size every year and the challenges were we wanted to get the right answers right and uh we had to figure out what that meant we had to implement that and we had to make it all uh efficient and uh we had to keep on testing and seeing if we were delivering good answers and now when you say good answers it means whatever people are typing in in terms of keywords in terms that kind of thing that the the results they get are ordered by the desirability for them of those results like they're like the first thing they click on will likely be the thing that they were actually looking for right one of the metrics we had was focused on the first thing uh some of it was focused on the whole page so it was focused on you know top three or so so we looked at a lot of different metrics for for how well we were doing and we broke it down into subclasses of you know maybe here's a type of uh of uh query that we're not doing well on then we try to fix that early on we started to realize that we were in an adversarial position right so we started thinking uh well we're kind of like the card catalog in the library right so the books are here and we're off to the side and we're just reflecting what's there and then we realized every time we make a change the webmasters make a change and it's uh game theoretic and so we had to think not only of is this the right move for us to make now but also if we make this move what's the counter move going to be is that going to get us into a work worst place in which case we won't make that move we'll make a different move and did you find i mean i assume with the popularity and the growth of the internet that people were creating new content so you're almost helping guide the creation yeah so that's certainly true right so we we definitely changed uh the structure of the network right so if you think back you know in the in the very early days uh larry and sergey had the page rank paper and john kleinberg had this uh hubsan authorities model which says the web is made out of these uh hubs which will be my page of cool links about dogs or whatever and people would just list links uh and then there'd be authorities which were the ones uh the page about dogs that most people link to that doesn't happen anymore people don't bother to say my page of cool links because we took over that function right so so we changed the way that worked did you imagine back then that the internet would be as massively vibrant as it is today i mean it was already growing quickly but it's just another i i don't know if you've ever if today if you sit back and just look at the internet with wander the amount of content that's just constantly being created constantly being shared unemployed yeah it's uh it's always been surprising to me i guess i'm not very good at predicting the future in the future okay and i remember you know being a graduate student in in 1980 or so and uh you know we had the arpanet and then there was this uh proposal to uh commercialize it and have this internet and this uh uh crazy senator gore thought that might be a good idea yeah and i remember thinking oh come on you can't you can't expect a commercial company to understand this technology they'll never be able to do it yeah okay we can have this dot-com domain but it won't go anywhere so i was wrong al gore was right at the same time the nature of what it means to be a commercial company has changed too so google yeah isn't that it's founding is different than uh you know what companies were before i think right so there's all these uh business models that are so different than what was possible back then so in terms of predicting the future what do you think it takes to build a system that approaches human level intelligence you've talked about of course that we you know we shouldn't be so obsessed about creating human level intelligence just create systems that are very useful for humans but what do you think it takes to uh to uh yeah approach that level right so certainly i don't think human level intelligence is one thing right so i think there's lots of different tasks lots of different capabilities i also don't think uh that should be the goal right so i you know i wouldn't want to create a uh calculator that could do multiplication at human levels right that would be a step backwards and so for many things we should be aiming far beyond human level for other things maybe human level is a good level to aim at and for others we say well let's not bother doing this because we already have humans can take on those tasks so as you say i like to focus on what what's a useful tool and and in some cases being on human level is an important part of crossing that threshold to make the tool useful so we see in things like these uh personal assistants now that you get either on your phone or on a speaker that sits on the table you want to be able to have a conversation with those and and i think as an industry we haven't quite figured out what the right model is for what these things can do and we're aiming towards well you just have a conversation with them the way you can with the person right but we haven't delivered on that model yet right so you can ask it what's the weather you can ask it play some nice songs uh and uh you know five or six other things and then you run out of stuff that it can do in terms of a deep meaningful connection so you've mentioned the movie her as one of your favorite ai movies do you think it's possible for a human being to fall in love with an ai system ai assistant as you mentioned so taking this big leap from uh what's the weather to you know having a deep connection yeah i i think uh as people that's what we love to do and uh i was at a a showing of her where we had a panel discussion and and somebody asked me uh what other movie do you think her is similar to and my answer was uh life of brian which which is not a science fiction movie uh but both movies are about wanting to believe in something that's not necessarily real yeah by the way for people don't know it's monty python yeah yeah that's been brilliantly put right so i mean i think that's just the way we are we we want to trust we want to believe we want to fall in love and uh it doesn't necessarily take that much right so you know my kids uh fell in love with their teddy bear right and the teddy bear was not very interactive right so that's all us yeah pushing our feelings onto our devices and our things and i think that that's what we like to do so we'll continue to do that so yeah as human beings will long for that connection and just ai has to uh do a little bit of work to uh to catch us in the other end yeah and certainly you know if you can get to uh dog level a lot of people have invested a lot of uh love in their pets and their pets some some people as i've been told in working with autonomous vehicles have invested a lot of love into their inanimate cars yeah so it really doesn't take much so what is a good test to linger on a topic that may be silly or a little bit philosophical what is a good test of intelligence in your view is natural conversation like in the touring test a good a good test put another way what would impress you yeah if you saw a computer do it these days yeah i mean i get impressed all the time right but like really impressive you know go playing uh starcraft playing uh those are all pretty cool you know and i think uh sure conversation is important i think uh you know we sometimes have these tests where it's easy to fool the system where you can have a chatbot that can have a conversation but you never uh it never gets into a situation where it has to be deep enough that uh it really reveals itself as being intelligent or not i think uh you know turing suggested that uh but i think if he were alive he'd say you know i didn't really mean that seriously right yeah and i think uh and you know this is just my opinion but but i think turing's point was not that uh this test of conversation is a good test i think his point was having a test is the right thing so rather than having the philosopher say oh no ai is impossible you should say well we'll just have a test and then the result of that will will tell us the answer and it doesn't necessarily have to be a conversation test that's right and coming up a new better test as the technology evolves is probably the right way do you worry as a lot of the general public does about not a lot but some vocal uh part of the general public about the existential threat of artificial intelligence so looking farther into the future as you said most of us are not able to predict much so when shrouded in such mystery there's a concern of well you think you start thinking about worst case is that something that occupies your mind space much so i certainly think about uh threats i think about uh dangers uh and i think uh any new technology uh has positives and negatives and if it's a powerful technology it can be used for bad as well as for good so i'm certainly not worried about uh the robot apocalypse the terminator type scenarios i am worried about change in employment and uh are we going to be able to react fast enough to deal with that i think we're you know we're already seeing it today where a lot of people are are disgruntled about uh the way income inequality is working and uh and automation could help accelerate those kinds of problems i see powerful technologies can always be used as weapons uh whether they're robots or drones or whatever uh some of that we're seeing due to ai a lot of it you don't need ai and i don't know what's uh what's a worse threat if it's an autonomous drone or it's uh crispr technology becoming available or we have lots of threats to face and some of them involve ai and some of them don't so the threats that technology presents are you for the most part optimistic about technology also alleviating those threats so creating new opportunities or protecting us from the more detrimental effects of these things yeah i don't know it again it's hard to predict the future and uh yes as a success society so far we've survived the nuclear systems and other things of course uh only societies that have survived are having this conversation so uh maybe that's a survivorship bias there yeah what problem stands out to you as exciting challenging impactful to work on in the near future for yourself for the community in broadly so i you know we talked about these uh assistance in conversation i think that's a great area i think combining uh common sense reasoning uh with the power of data is a a great area in which application in in conversation relation or just broadly just in general yeah as a programmer i'm interested in uh programming tools both in terms of uh you know the current systems we have today with with tensorflow and so on can we make them much easier to use for broader uh class of people and also can we apply uh machine learning to the more traditional type of programming right so you know when you go to google and you uh type in a query and you spell something wrong it says did you mean and the reason we're able to do that is because lots of other people made a similar error and then they corrected it we should be able to go into our code bases and our bug fix spaces and when i type a line of code it should be able to say did you mean such and such if you type this today you're probably going to type in this bug fix tomorrow yeah that's a really exciting application of uh almost uh an assistant for the coding programming experience yeah at every level so i think i could safely speak for the entire ai community first of all for uh thank you for the amazing work you've done uh certainly for the amazing work you've done with uh ai a modern approach book yep i think we're all looking forward very much for the fourth edition and then the fifth edition and so on so uh peter thank you so much for talking today yeah thank you pleasure you