Charles Isbell and Michael Littman: Machine Learning and Education | Lex Fridman Podcast #148
the following is a conversation with charles isbell and michael littman charles is the dean of the college of competing at georgia tech and michael is a computer science professor at brown university i've spoken with each of them individually on this podcast and since they are good friends in real life we all thought it would be fun to have a conversation together quick mention of each sponsor followed by some thoughts related to the episode thank you to athletic greens the all-in-one drink that i start every day with to cover all my nutritional bases ate sleep a mattress that cools itself and gives me yet another reason to enjoy sleep master class online courses from some of the most amazing humans in history and cash app the app i use to send money to friends please check out the sponsors in the description to get a discount and to support this podcast as a side note let me say that having two guests on the podcast is an experiment that i've been meaning to do for a while in particular because down the road i would like to occasionally be a kind of moderator for debates between people that may disagree in some interesting ways if you have suggestions for who you would like to see debate on this podcast let me know as with all experiments of this kind it is a learning process both the video and the audio might need improvement i realized i think i should probably do three or more cameras next time as opposed to just two and also try different ways to mount the microphone for the third person also after recording this intro i'm going to have to go figure out the thumbnail for the video version of the podcast since i usually put the guest's head on the thumbnail and now there's two heads and two names to try to fit into the thumbnail it's a kind of bin packing problem which in uh theoretical computer science happens to be an np hard problem whatever i come up with if you have better ideas for the thumbnail let me know as well and in general i always welcome ideas how this thing can be improved if you enjoy it subscribe on youtube review it with five stars on apple podcast follow on spotify support on patreon or connect with me on twitter and lex friedman and now here's my conversation with charles isbell and michael littman you'll probably disagree about this question but what is your biggest would you say disagreement about either something profound and very important or something completely not important at all i don't think we have any disagreements at all ah i'm not sure that's true we walked into that one didn't we yeah so one thing that you sometimes mention is that and we did this one on air too as it were whether or not machine learning is computational statistics it's not but it is well it's not and in particular and more importantly it is not just computational statistics so what's missing in the picture what all the rest of it what's missing that which is missing oh because well you can't be wrong now well it's not just the statistics he doesn't even believe this we've had this conversation before if it were just the statistics then we would be happy with where we are but it's not just the statistics that's why it's computational statistics or if it were just the computation i agree that machine learning is not just statistics it is not just this we can agree on that nor is it just computational statistics it's computational statistics it is computational what is the computational and computational statistics does this take us into the realm of computing it does but i think perhaps the way i can get him to admit that uh he's wrong is that it's about rules it's about rules it's about symbols it's about all these other things statistics it's not about rules i'm going to say statistics is about rules but it's not just the statistics right it's not just a random variable that you choose and you have a probability i think you have a narrow view of statistics okay well then what would be the broad view of statistics that would still allow it to be statistics and not say history that would make computational statistics okay well okay so i i had my first sort of research mentor a guy named tom landauer taught me to do some statistics right sure and and i was annoyed all the time because the statistics would say that what i was doing was not statistically significant and i was like but but but and basically what he said to me is statistics is how you're going to keep from lying to yourself which i thought was really deep it is a way to keep yourself honest in a particular way i agree with that yeah and so you're trying to find rules i'm just kind of bringing back to rules wait wait wait could you possibly try to define rules even regular statisticians non-computational statisticians do spend some of their time evaluating rules right applying statistics to try to understand is this you know is this does this rule capture this does this not capture i mean like hypothesis testing kind of or like confidence intervals like like have like more like hypothesis like i feel like the word statistic literally means like a summary like a number that summarizes other numbers right but i think the field of statistics actually applies that idea to things like rules to understand whether or not a rule is valid the software engineering statistics no programming languages statistics no because i think there's a very it's useful to think about a lot of what ai and machine learning is or certainly should be as software engineering uh as programming languages just if to put it in language that you might understand in the hyper parameters beyond the problem the hyper parameters has too many syllables for me to understand the hyperparameters of uh that's better that goes around it right it's the decisions you choose to make it's the metrics you choose to use it's the loss you want to say the practice of machine learning is different than the practice of statistics like the things you have to worry about and how you worry about them are different therefore they're different right at a very little i mean at the very least it's that that much is true it doesn't mean that statistics computational or otherwise aren't important i think they are i mean i do a lot of that for example but i think it goes beyond and i think that we could think about game theory in terms of statistics but i don't think it's very as useful to do i mean the way i would think about it or a way i would think about it is this way chemistry is just physics but i don't think it's as useful to think about chemistry as being just physics it's useful to think about it as chemistry the level of abstraction really matters here so i think it is there are contexts in which it is useful that way right so finding that connection is actually helpful and i think that's when i when i emphasize the computational statistics thing i think i think i want to befriend statistics and not absorb them here's the here's the a way to think about it beyond what i just said right so what would you say and i want you to think back to a conversation we had a very long time ago what would you say is the difference between say the early 2000's icml and what we used to call nips nerfs was there a difference a lot of the particularly on the machine learning that was done there icmo was around that long oh yeah so iclear is the new conference newish uh yeah i guess so and i see him i was around the 2000 oh i see male predates that i well i think my most cited icml paper is from 94. yeah michael knows this better than me because of course he's significantly older than i am but the point is yeah what is the difference what was the difference between icml and nureps in the late 90s early 2000s i don't know what everyone else's perspective would be but i had a particular perspective at that time which is i felt like icml was more of a of a computer science place and that nips nerfs was more of an engineering place like the kind of math that happened at the two places as a computer scientist i felt more comfortable with the icml math and the nurbs people would say that that's because i'm dumb and that's such an engineering thing to say so i agree with that part of it but i do a little differently actually i had a nice conversation with tom dietrich about this in public on twitter just a couple days ago i put it a little differently which is that icml was machine learning done by uh computer scientists and uh nurbs was machine learning done by computer scientists trying to impress statisticians which was weird because it's the same people at least by the time i started paying attention but it just felt very very different and i think that that perspective of whether you're trying to impress the statisticians or you're trying to impress the programmers is actually very different and has real impact on what yeah what you choose to worry about and what kind of uh outcomes you come to so i think it really matters in computational statistics is a means to an end it is not an end in some sense um and i think that really matters here in the same way that i don't think computer science is just engineering or just science or just math or whatever but okay so i'd have to now agree that now we agree on everything yes yes the important thing here is that you know my opinions may have changed but not the fact that i'm right i think is what what we just came to right now my opinions may have changed and not the fact that i'm wrong that's right i lost me i'm not i think i lost myself there too but anyway this happens to us sometimes we're sorry how does neural networks change this just to even linger on this topic change this idea of statistics how big of a pi statistics is within the machine learning thing like because it sounds like hyper parameters and also just the role of data you know this people are starting to use this terminology software 2.0 which is like the act of programming as a as a like you're a designer in the hyperparameter space of neural networks and you're also the collector and the organizer and the cleaner of the data and that's part of the programming uh so how did on the versus icml topic what's the role of neural networks and redefining the size and the role of machine learning i can't i can't wait to hear what michael thinks about this but um i would add one well but that's true i'll force myself to i think the the there's one thing i would add to your description which is the kind of software engineering part is what does it mean to debug for example but this is a difference between uh the kind of computational statistics view of machine learning and the computational view of machine learning which is i think one is worried about the equation as it were and by the way this is not a value judgment i just think it's about perspective but the kind of questions you would ask when you start asking yourself what does it mean to program and develop and build the system it's a very computer sciencey view of the problem i mean when if you get on data science twitter and econ twitter you actually hear this a lot with the uh you know the economist and the data scientist complaining about the machine learning people well you know it's just statistics and i don't know why they don't don't see this but they're not even asking the same questions they're not thinking about it as a kind of programming problem and i think that that really matters just asking this question i actually think it's a little different from programming and hyper parameter space and sort of collecting the data but i do think that that immersion really matters so i'll give you a quick a quick example the way i think about this so i teach machine learning michael and i have co-taught a machine learning class which has now reached i don't know 10 000 people at least over the last several years or somewhere there's abouts and my machine learning assignments are of this form so the super the first one is something like implement these five algorithms you know k n and s you know svms and boosting and decision trees and neural networks and maybe that's it i can't remember and when i say implement i mean steal the code i am completely uninterested you get zero points for getting the thing to work i don't want you spending your time worrying about uh getting the corner case right of you know what happens when you are trying to normalize distances and the points on the thing and so you divide by zero i'm not interested in that right steal the code however you're going to run those algorithms on two data sets the data sets have to be interesting what does it mean to be interesting well data says interesting if it reveals differences between algorithms which presumably are all the same because they can represent whatever they can represent and two data sets are interesting together if they show different differences as it were and you have to analyze them you have to justify their interestingness and you have to analyze in a whole bunch of ways but all i care about is the data in your analysis not the programming and i occasionally end up in these long discussions with students well i don't really i copy and paste the things that i've said the other 15 000 times it's come up which is they go but the only way to learn really understand is to code them up which is a very programmer software engineering view of the world if you don't program it you don't understand it which is by the way i think is wrong in a very specific way but it is a way that you come to understand because then you have to wrestle with the algorithm but the thing about machine learning is it's not just sorting numbers where in some sense the data doesn't matter what matters is well does algorithm work on these abstract things and one less than the other in machine learning the data matters it does it matters more than almost anything and not everything but almost anything and so as a result you have to live with the data and don't get distracted by the algorithm per se and i think that that focus on the data and what it can tell you and what question it's actually answering for you as opposed to the question you thought you were asking is a key and important thing about machine learning and is a way that computationalists as opposed to statisticians bring a particular view about how to think about the process the statisticians by contrast bring i i think i'd be willing to say a better view about the kind of formal math that's behind it and what an actual number ultimately is saying about the data and those are both important but they're also different i didn't really think of it this way is to build intuition about the role of data the different characteristics of data by having two data sets that are different and they reveal the differences in the differences that's that's a really fascinating that's a really interesting educational approach the students love it but not right away no they love it later i love it at the end not at the beginning not even not even immediately after i feel like there's a deep profound lesson about education there yeah that uh you can't listen to students about whether what you're doing is the right or the wrong thing well as a wise uh michael litman once said to me about children which i think applies to teaching is you have to give them what they need without bending to their will and students are like that you have to figure out what they need you're a curator your whole job is to curate and to present because on their own they're not going to necessarily know where to search so you're providing pushes in some direction and learn space and you have to give them what they need in a way that keeps them engaged enough so that they eventually discover what they want and they get the tools they need to go and learn other things what's your view let me put on my russian hat which believes that life is like russian hats by the way if you have one i would like those are ridiculous yes but in a delightful way but sure what do you think is the role of uh we talked about balance a little bit what do you think is the role of hardship in education like i think the biggest things i've learned like what made me fall in love with math for example is by being bad at it until i got good at it so like like struggling with a problem which increased the level of joy i felt when i finally figured it out and it always felt with me with teachers especially modern discussions of education how can we make education more fun more engaging more all those things or from my perspective it's like you're maybe missing the point that education that life is suffering education is supposed to be hard and that actually what increases the joy you feel when you actually learn something is that ridiculous do you like to see your students suffer okay so this may be a point where we differ i'd suspect not i'm gonna do go on well what would your answer be i wanna hear you first okay well i would i was gonna not answer the question do you know what this dude is i wasn't gonna hear them suffering no no no no no i was i was gonna say that there's i think there's a distinction that you can make in the kind of suffering right so i think you can be in a mode where you're you're suffering in a hopeless way versus you're suffering in a hopeful way right where you're like you can see that if you that you still have you can still imagine getting to the end right and as long as people are in that mindset where they're struggling but it's not a hopeless kind of struggling that's that's productive i think that's really helpful but it's struggling like if you break their will if you leave them hopeless no that don't sure some people are gonna whatever lift themselves up by their bootstraps but like mostly you give up and certainly it takes the joy out of it and you're not going to spend a lot of time on something that brings you no joy so it's it's it is a bit of a delicate balance right you have to thwart people in a way that they still believe that there's a way through right so that's a that we strongly agree actually so i think well first off struggling and suffering aren't the same thing right being poetic oh no no i actually appreciate the poetry and i one of the reasons i appreciate it is that they are often the same thing and often quite different right so you can struggle without suffering you can certainly suffer and suffer suffer pretty easily you don't necessarily have to struggle to suffer so i think that you want people to struggle but that hope matters you have to they have to understand that they're going to get through it on the other side and it's very easy to confuse the two i actually think brown university has a very just philosophically has a very different take on the relationship with their students particularly undergrads from say a place like georgia tech which is which universities better uh well i have my opinions on that i mean remember charles said it doesn't matter what the facts are i'm always right the correct thing is that it doesn't matter they're different um but clearly he went to a school like the school where he is as an undergrad i went to a school specifically the same school though it was it changed a bit in the in the intervening years brown or georgia tech no i was talking about georgia tech and i went yeah and i went to an undergrad place that's a lot like the place where i work now and so it does seem like we're more familiar with these models there's a similarity between brown and yellow yeah there's a i think that i think they're quite similar yeah and duke duke has some similarities too but it's got a little southern draw you've kind of worked here you sort of worked at universities that are like the places where you learned and the same would be true for me are you uncomfortable uh venturing outside the box is that what you're saying journeying out what i'm saying yeah charles is definitely he only goes to places that have institute in the name right it has worked out that way well academic places anyway well no i was a visiting scientist at upenn or visiting visiting something at upenn oh wow i just i just understood your joke which one five minutes later i like to set these sort of time bombs the institute is in the uh uh that charles only goes to places that have institute in the name so i guess georgia i forget that georgia tech is georgia institute of technology the number of people who refer to it as georgia tech university is large and incredibly irritating that's one of the few things that generally gets under my schedule but like schools like georgia tech and mit have as part of the ethos like there is i want to say there's a there's an abbreviation that someone taught me like i htfp something like that like there's a there's a there's an expression which is basically i hate being here which they say so proudly and that is definitely not the ethos at brown like brown is there's a little more pampering and empowerment and stuff and it's not like we're gonna crush you and you're gonna love it so yeah i think there's a i think the ethos are different mm-hmm that's interesting yeah we had drone proofing what's that trump graduate from georgia tech this is a true thing feel free to look it up uh if you a lot of schools have this by the way no actually georgette was barely the first brandeis has it had it i feel like georgia tech was the first in the look first of all it was it was the first time i think um had the first time stop that first masters in computer science actually right online masters well that too but way back in the 60s um nsf yeah yeah you're the first information and computer science master's degree in the country um but the uh georgia tech it used to be the case in order to graduate from georgia tech uh you had to take a drown proofing class where effectively they threw you water tied you up if you didn't drown you got to graduate i believe so there were certainly versions of it but i mean luckily they ended it just before i had to graduate because otherwise would have never graduated it wasn't going to happen uh i want to say 84 or 83 someone around then they they ended it but uh yeah you used to have to prove you could tread water for some ridiculous amount of time are you two yeah you couldn't graduate no it was more than two hours two minutes okay it was in a bathtub it was in a pool but it was a real thing but that idea that you know push you fully clothed yeah fully clothed i don't think i bet it was that and not tied up because like who needs to learn how to swim when you're tied nobody but who needs to learn when to swim when you're actually falling into the water dressed that's a real thing i think your facts are getting in the way with a good story oh that's fair that's fair i didn't think all right so they didn't tell you what the narrative mattered but whatever it was you had to it was called drown proofing for a reason the point of the story michael uh is struggle it it's well no but that's good it doesn't bring it back to struggle that's a part of what georgia tech has always been and we struggle with that by the way uh about what we want to be as things go but you you sort of how much can you be pushed without breaking and you come out of the other end stronger right there there's this saying we said when i was an undergrad there which is georgia tech building tomorrow the night before right kind of idea that you know give me something impossible to do and i'll do it in a couple of days because that's what i just spent the last four or five or six years that ethos definitely stuck to you having now done a number of projects with you you definitely will do it the night before that's not entirely true there's nothing wrong with waiting until the last minute the secret is knowing when the last minute is right that's brilliant that's brilliantly put yeah that yeah that's that is a definite charles statement that i am trying not to embrace and i appreciate that because you helped move my last minute that's the social construct that we converge together what the definition of last minute is and we we figure that out all together in fact mit you know i'm sure a lot of universities have this but mit has like mit time that yeah everyone has always agreed together that there is such a concept and everyone just keeps showing up like 10 to 15 to 20 depending on the department late to everything so there's like a weird drift that happens it's kind of fascinating yeah we're five minutes five minutes in fact the classes will say you know well this is no longer true actually but it used to be a class was started eight but actually started 805 yeah it ends at nine actually ends at 8 55. uh everything's five minutes off and nobody expects anything to start until five minutes after the half hour or whatever it is it still exists it hurts my head well let's rewind the clock back to the 50s and 60s when you guys met how did you i'm just kidding i don't know but what can you tell the story of how you met so you've like the internet and the world kind of knows you as as as connected in some ways in terms of education of teaching the world that's that's like the public facing thing but how did you as human beings and as collaborators meet i think there's two stories one is how we met and the other is how we got to know each other i'm not gonna say fellaini i'm gonna say that we came to understand that we had some common something yeah it's funny because on the surface i think we're we're different in a lot of ways but there's something yeah i mean that's just consonant there you go afternoon so i will tell the story of how we met and i'll let michael tell the story of how we okay all right okay so here's how we met um i was already at that point it was 18t labs there's a long interesting story there but anyway i was there and uh michael was coming to interview he was a professor at duke at the time but decided for reasons that he wanted to be in new jersey uh and so that would mean uh bell lab slash att labs uh and we were doing interview interviews very much like academic interviews uh and so i had to be there uh we all had to meet with him afterwards and so on one on one but it was obvious to me that he was gonna be hired like no matter what because everyone loved him they were just talking about all the great stuff he did and oh he did this great thing and you just won something at triple a i think or maybe you got 18 papers in triple either but i got the best paper award at your play for the crosswords right exactly so that it all happened and everyone was going on and on and on about actually tinder was saying incredibly nice things about you really yes so he can be very grumpy yes that's very that's nice to hear he was grumpily saying very nice things oh that's that makes sense and that does make sense so you know so it was going to come so why were we why was i meeting him i had something else i had to do i came here what it was yeah it probably involved commenting he remembers meeting me as inconveniencing his afternoon so he came so eventually came to my office i was in the middle trying to do something i can't remember what and he came and he sat down and for reasons that are purely accidental despite what michael thinks my desk at the time was set up in such a way that had sort of an l shape and the chair on the outside was always lower than the chair that i was in and you know the kind of point was the only reason i think that was on purpose is because you told me it was on purpose i don't remember that anyway the thing is that you know it kind of his guest chair was really low so that he could yeah he could look down at everybody the idea was just to simply create a nice environment that you were asking for a mortgage and i was going to say no that was a very simple idea here anyway so we sat there and we just talked for a little while and i think he got the impression that i didn't like him that wasn't true strongly the talk was really good by the way it was terrible and after right after the talk i said to my host michael kearns who ultimately was my boss i'm a huge fan i'm a friend and a huge fan of michael yeah yeah he is a remarkable person um i i after my talk today i went into this i went back at ball he's good at that basketball no but basketball racquetball squash which is not racquetball yes squash no and i hope you you hear that michael you mean like your parents as a game not his skill level because i'm pretty sure he's all right there's some competitiveness there but the point is that it was like the middle of the day i had full day of interviews like i met with people but then in the middle of the day i gave a job talk and then um and then there was going to be more interviews but i i pulled michael aside and i said i think it's in both of our best interests if i just leave now because that was so bad that it's just be embarrassing if i have to talk to any more people like you look bad for having invited me like it's just let's just forget this ever happened so i don't think the talk went well it's one of the most michael littman set of sentences i think i've ever heard he did great or at least everyone knew he was great so maybe it didn't matter i was there i remember the talk and i remember him being very much the way i remember him now in any given week so it was good and we met and we talked about stuff he thinks i didn't like him but because he was so grumpy must been the chair thing the chair thing and the low voice i think the like obviously and that like that like slight like skeptical look yeah i have no idea what you're talking about well i probably didn't have any idea what you were talking about anyway i liked him he asked me questions i answered questions i felt bad about myself it was a normal day then he left and then he left and that's how you tell me can we take it and then i got hired and i was in the group can we take a slight tangent on that on this topic of it sounds like uh maybe you could speak to the bigger picture it sounds like you're quite self-critical who charles no you oh i think i can i can do better i can do better i'll try me again i'll i'll do better yeah that was like a like a three out of ten responses so let's try to work it up to five and six uh you know i remember uh marvin minsky said uh on on a video interview something that the key to success in academic research is to hate everything you do for some reason i think i followed that because i hate everything he's done [Laughter] uh it's a good line that's a success maybe that's a keeper but um but do you do find that resonates with you at all in how you think about talks and so on i would say it differently it's not really that's such an mit view of the world though so i remember i i remember talking about this when uh as a student you know you were basically told uh i will clean it up for the purpose of the podcast um uh my work is crap my work is crap my work is crap my work is crap then you like go to a conference or something like everybody else's work is crap everybody else is working crap and you feel better and better about it yeah uh relatively speaking and then you sort of keep working on it i don't hate my work that resonates with me yes i've never hated my work but i have i have been dissatisfied with it and i think being dissatisfied being okay with the fact that you've taken a positive step the derivative is positive maybe even the second derivative is positive that's important because that's a part of the the hope right but you have to but i haven't gotten there yet if that's not there that i haven't gotten there yet then you know it's hard to it's hard to move forward i think so i buy that which is a little different from hating everything that you do yeah i mean there's there's things that i've done that i like better than i like myself so it's separating me from the work essentially so i think i am very critical of myself but sometimes the work i'm really excited about and sometimes i think it doesn't happen right away so i found the work that i've liked that i've done most of it i liked it in retrospect more when i was far away from it in time i have to be fairly excited about it to get done no excited at the time but then happy with the result or but years later or even i might go back you know what that actually turned out to be yeah that turned out to matter or oh gosh it turns out i've been thinking about that it's actually influenced all the work that i've done since without realizing it but that guy was smart yeah that guy had a future yeah yeah he's going places i think there's so yeah so i think there's something to it i think there's something to the idea you've got to you know hate what you do but it's not quite hate it's just being unsatisfied and different people motivate themselves differently i don't happen to motivate myself with self-loathing i happen to motivate myself so you're able to sit back and be proud of in retrospect of the work you've done well and it's easier when you can connect with other people because then you can be proud of them a lot of the people yeah and then the questions you can still safely hate yourself it's a win-win michael or at least win lose which is what you're looking for oh wow there's so many brilliant lines in this there's levels uh so how did you actually meet me yeah so my the way i think about it is because we didn't do much research together at 18t but um but then we all got laid off so so that was that by the way i decided to interrupt but that was like one of the most magical places historically speaking they did not appreciate what they had and how do we uh i feel like there's a profound lesson in there too uh how do we get it like what was why was it so magical is just the coincidence of history or is there something special some really good managers and people who really believed in machine learning as this is going to be important um let's get the the people who are thinking about this in creative and and insightful ways and put them in one place and stir yeah but even beyond that right it was it was bell labs at its heyday and even when we were there which i think was past it to be clear he's gotten to be at bell labs i never got to be at bell labs i joined after that yeah i should have been 91 as a grad student so i was there for a long time um every summer except twice i worked for companies that had just stopped being better labs right bell core and then att labs so about labs was several locations or for the for the research or is it what like jerseys are involved somehow they're all in jersey yeah they're all over the place but they're in a couple places murray hill was the bell labs um so you you had you had an office in mary hill at one point in your career yeah and i i played ultimate frisbee on the cricket pitch at bell labs at murray hill uh and then it became 18t labs when split off with loose during what we called uh tri-vestiture supposedly better than michael koren's ultimate frisbee yeah oh yeah okay but i think that one's not boasting i think that i think charles plays a lot of ultimate and i don't think mike i was yes but but that wasn't the point the point is yes yes sorry okay i have played on a championship winning ultimate frisbee team or whatever ultimate team with charles so i know how good he is he's really good how good i was anyway when i was younger but the thing is i know how young he was when he was yeah that's true that was true so much younger than now he's old enough yeah i'm older michael is a much was a much better basketball player than i was michael kearns yes no not michael i'm very clear so you don't know how terrible i am but you have a probably pretty good guess that you're not as good as michael kearns he's tall and and he cared about it very outlet he's very good he's probably competitive i love hanging out with michael anyway but we were talking about something else although i no longer remember what it was what were we talking about but also labs so so uh this was kind of cool about what was magical about it the first thing you have to know is that bell labs was an arm of the government right because att was an army of government it was a monopoly uh and you know every month you paid a little thing on your phone bill which turned out was a tax for like all the research that bell labs was doing and you know they invented transistors and the laser and whatever else is that big bang or whatever the cosmic background radiation yeah they did all that stuff they had some amazing stuff with directional microphones by the way i got to go in this room um where they they had all these panels and everything and we would talk and one another and he moved some panels around and then he would have me step two steps to the left and i couldn't hear a thing he was saying because nothing was bouncing off the walls and then he would shut it all down and you could hear your heartbeat yeah deeply disturbing to hear your heart beat you can feel it i mean you can feel it now there's so much all this sort of noise around anyway bill labs is about pure research it was a university in some sense the purest sense of a university but without students so it was all the faculty working with one another and students would come in to learn they would come in for three or four months you know during the summer and they would go away but it was just this kind of wonderful experience i could walk out my door in fact i would often have to walk out my door and deal with rich sutton and michael kearns yelling at each other about whatever it is they were yelling about the proper way to prove something or another and i could just do that and dave mcallister and evan and peter stone and and all of these other people including satinder and then eventually michael and it was just a place where you could think thoughts and it was okay because so long as once every 25 years or so somebody invented a transistor it paid for everything else you could afford to take the risk and then when that all went away it became harder and harder and harder to justify it as far as the folks who were very far away were concerned and there was such a fast turnaround among middle management on the atnt side that you never had a chance to really build the relationship at least people like us didn't have a chance to to build relationships so when the diaspora happened um it was amazing right yeah everybody left and i think everybody ended up at a great place and made a huge made a continued to do really good work with with machine learning but it was a wonderful place and people will ask me you know what's the best job you you've ever had and as a professor anyway the answer that i would give is um well probably bell labs in some very real sense and i would never have a job like that again because bell labs doesn't exist anymore and you know microsoft research is great and google does good stuff and you can pick ibm you can tell if you want to but bell labs was magical it was around for it was an important time and it represents a a high water mark in in basic research in the u.s is there something you could say about the physical proximity and the chance collisions like we live in this time of the pandemic where everyone is maybe trying to see the silver lining and accepting the remote nature of things is is there one of the things that people like faculty that i talk to miss is the the procrastination like the chance to like everything is about meetings that are supposed to be there's not a chance to just uh you know talk about comic book or whatever like go into discussion that's totally pointless so it's funny you say this because that's how we met matt it's exactly that so i'll let michael say that but i'll just add one thing which is just that uh you know research is a social process and it helps to have random social interactions even if they don't feel social at the time that's how you get things done one of the great things about the a lab when i was there i don't quite know what it looks like now once they moved buildings but we had entire walls that were whiteboards and people would just get up there and they were just right and people would walk up and you'd have arguments and you'd explain things to one another and you got so much out of the freedom to do that you had to be okay with people challenging every freaking word you said which i would sometimes find deeply irritating but most of the time it was it was quite useful but the sort of pointlessness and the interaction was in some sense the point at least for me yeah i mean you i think offline yesterday i mentioned josh tannenbaum and he's very much he put he's a man he's such an inspiration in in the child like way that he pulls you in on any topic it doesn't even have to be about machine learning it could or or the brain he'll just pull you into a closest writable surface which is uh still you can find whiteboards at mit everywhere and and just like uh like basically cancel all meetings and talk for a couple hours about some some aimless thing and it it feels like the whole world the time space continuum kind of warps and that becomes the most important thing and then it's just it's so true it's it's definitely something worth missing in this in this world where everything's remote there's some magic to the physical presence whenever i wonder myself whether mit really is as great as i remember it i just go talk to josh yeah you know that's funny is there's a few people in this world that carry the the best of what particular institutions stand for right and it's uh it's josh i mean i i don't i my guess is he's unaware of this that's the point that the masters are not aware of their mastery so how do we all meet yes but but first a tangent no how did you meet me so i'm not sure what you were thinking of but my when it started to dawn on me that maybe we had a longer-term bond was after we all got laid off and you had decided at that point that there we were still paid we were given an opportunity to like do job search and kind of make a transition but it was clear that we were done and i would go to my office to work and you would go to my office to keep me from working that was that was my recollection of it and you had decided that there was no really no point in working for the company because the company our relationship with the company was was done yeah but remember i felt that way beforehand it wasn't about the company it was about the set of people there doing really cool things and it always always been that way but we were working on something together oh yeah yeah that's right oh so at the very end we all got laid off but then our boss came to our boss's boss came to us because our boss was michael kearns and he had jumped ship brilliantly like perfect timing like things like right before the ship was about to sink he was like gotta go and and and landed perfectly because michael kearns because michael king and um leaving the rest of us to go like this is fine and then it was clear that wasn't fine and we were all toast so we had this sort of long period of time but then our boss figured out okay wait maybe we can save a couple of these people if we can have them do something really useful and uh the useful thing was we were going to make a basically an automated assistant that could help you with your calendar you could like tell it things and it would it would respond appropriately it would just kind of integrate across all sorts of your personal information and so me and charles and peter stone were this were set up as the crack team to actually solve this problem uh other people maybe were too theoretical that they thought and and but we could actually get something done so we sat down to get something done and there wasn't time and it wouldn't have saved us anyway and so it all kind of went downhill but the interesting i think coda to that is that our boss's boss is a guy named ron brockman and he when he left at t because we were all laid off he went to darpa started up a program there that became kalo which is the program from which siri sprung which is a digital assistant that helps you with your calendar and a bunch of other things um it really you know in some ways got its start with me and charles and peter trying to implement this vision that ron brockman had that he ultimately got implemented through his role at darpa so when i'm trying to feel less bad about having been laid off from what is possibly the greatest job of all time i think about well we kind of helped birth siri that's something and he did other things too but the we got to spend a lot of time in his office and talk about we got to spend a lot of time in my office yeah yeah yeah and so uh so then we went on our merry way everyone went to different places charles landed at georgia tech which was uh what he always dreamed he would do and so um that worked out well yeah um i came up with a saying at the time which is luck favors the charles it's kind of like luck favors the prepared but charles like like he'd wish something and then it would basically happen just the way he wanted it was it was inspirational to see things go that way things worked out and we stayed in touch and then um i think it really helped when you were working on i mean you kept me in the loop for things like threads and the work that you were doing at georgia tech but then when they were starting their online master's program he knew that i was really excited about moocs and online teaching and he's like i have a plan and i'm like tell me your plan he's like i can't tell you the plan yet because they were deep in in negotiations between georgia tech and udacity to make this happen and they didn't want it to leak so charles would kept teasing me about it but wouldn't tell me what was actually going on and eventually it was announced and he said i would like you to teach the machine learning course with me i'm like that can't possibly work um but it was a great idea and it was it was super fun it was a lot of work to put together but it was it was really great and was that the first time you thought about first of all was it the first time you got seriously into teaching i mean you know i'm trying to get the feeling right i'll tell you this is already after you jump to so like there's a little bit of jumping around in time yeah sorry about it there's a pretty big jump in time so like the moocs thing so charles got to georgia tech and he i mean maybe charles maybe this is a trick in 2002. he got to georgia tech in 2002 and um but then and worked on things like revamping the curriculum the undergraduate curriculum so that it had some kind of semblance of modular structure because computer science was at the time moving from a fairly narrow specific set of topics to touching a lot of other parts of of of intellectual life and the curriculum was supposed to reflect that and so um charles played a big role in in kind of redesigning that and then and for my and for my my labors i ended up his associate dean right he got to become an associate dean of in charge of educational stuff well this would be a valuable lesson if you're good at something uh they will give you responsibility to do more of that thing well until you don't show confidence don't show confidence if you well you know what the responsibility here's what they say yeah the reward for good work is more work the reward for bad work is less work which i don't know depending about what you're trying to do that week one of those is better than the other well one of the problems with the word work sorry to interrupt is that it's seems to be an antonym in this particular language we have the opposite of happiness but it seems like they're they're like that's one of you know we talked about balance it's uh it's always like work-life balance it always rubbed me the wrong way as a terminology i know it's just words right the opposite of work is play but yeah ideally work is play oh i can't tell you how much time i'd spend certainly i was about labs except for a few very key moments uh as a professor i would do this too i was just saying cannot believe they're paying me to do that um because it's fun it's something that i would i would do for a hobby if i could anyway uh so that sort of worked out i'm sure you want to be saying that when this is being recorded as a dean that is not true at all i need a raise yes but but i think here with with this that even though a lot of time passed you know michael and i talked almost every well we texted almost every day during the period charles at one point took me there was the icml conference the machine learning conference was in atlanta i was the chair the general chair of the conference charles was my publicity chair or something like that or something fundraising champion sure yeah um but he decided it'd be really funny if he didn't actually show up for the conference in his own home city uh so he didn't but he did at one point picked me up at the conference in his tesla and drove me to the atlanta mall and forced me to buy an iphone because he didn't like how it was to text with me and thought it would be better for him if i had an iphone the text would be somehow smoother and it was and it was and it is and his life is better and my life is better and so death but but it was yeah charles forcing me to get an iphone so that he could text me more efficiently i thought that was an interesting moment it works for me anyway so we kept talking the whole time and then eventually we did the we did the teaching thing and it was great and there's a couple of reasons for that by the way one is i really wanted to do something different like you've got this medium here people claim it can change things what's a thing that you could do in this medium that you could not do otherwise besides edit right i mean what could you do and and being able to do something with another person was that kind of thing it's very hard i mean you can take turns but teaching together having conversations is very hard right so that was a cool thing the second thing it gave me an excuse to do more stuff with him yeah i always thought he makes it sound brilliant um and it is i guess but it's at the time it really felt like i've got a lot to do charles is saying and it would be great if michael could teach the course and i could just hang out yeah just kind of coast on that well that's what the second class was more like that because the second time that was explicit because the first class it was at least half so the structure so that was that was kind of true yeah that was sort of true for 7642 which is the reinforcement learning class because that was really his class you started with reinforcement no we started with i did the intro machine learning 7641 uh which is supervised learning unsupervised learning and reinforcement learning and decision making cram all that in there the kind of assignments that we talked about earlier and then eventually about a year later we did a follow-on 7642 which is reinforcement learning and decision making the first class was based on something i'd been teaching at that point for well over a decade and the second class was based on something michael had been teachers actually i learned quite a bit teaching that class with him but he drove most of that but the first one i drove most it was all my material although i had stolen that material originally from slides i found online from michael who had originally stolen that material from i guess slides he found online probably from andrew moore because the jokes were the same anyway at least some of the at least when i found the slides some of the stuff yes every machine learning class taught in the early 2000's stole from andrew moore a particular joke or two the at least the structure now i did and he did actually a lot more with reinforcement learning and such and game theory and those kinds of things but you know we all sort of know this world no no no no i mean teaching that class the coverage was different than than what other people were starting most people were just doing supervised learning and maybe a little bit of you know clustering and whatnot but we took it all the way to a lot of it just comes from tom mitchell's book oh no yeah except well half of it comes from tom mitchell's book right i mean the other half doesn't this is what this is why it's all readings right because certain things weren't invented when tom wrote okay that's true right uh but it was it was quite good but there's a reason for that besides you know just i wanted to do it i wanted to do something new and i wanted to do something with him which is a realization which is despite what you might believe he's an introvert and i'm an introvert or i'm on the edge of it or being an introvert anyway but both of us i think um enjoy the energy of the crowd right there's something about talking to people and bringing them into whatever we find interesting that is empowering energy energizing or whatever and i found the idea of staring alone uh at a computer screen and then talking off of materials less inspiring than i wanted it to be and i had in fact done a mooc for udacity on algorithms and it was a week in a dark room talking at the screen writing on the little pad and i didn't know this was happening but they had watched the crew had watched some of the videos while like in the middle of this and they're like something's wrong you're you're sort of shutting down um and i think a lot of it was i'll make jokes and no one would laugh yeah and i felt like the crowd hated me now of course there was no crowd so like it wasn't rational yeah but it's little each time i tried it and i got no reaction it just was taking the the the energy out of my performance out of my presentation fantastic metaphor for grad school anyway by working together we could play off each other and have it and and have it keep the energy up because you can't you with you can't let your guard down for a moment with charles he'll just he'll just overpower you i have no idea what's wrong with but we would work really well together i thought and we knew each other so i knew that we could we could sort of make it work plus i was the associate dean so they had to do what i told him to do we had to do that we had to make it work and so it worked out very well i thought um well enough that we with great power comes great power that's right and we became smooth and curly and uh that's when we we we did the the um the uh overfitting thriller video yeah we took yeah yep that's a thing so what okay can we just like like uh smooth and curly where was that so okay so that happened it was completely spontaneous these are the nicknames you go by yeah so uh students call us he was he was lecturing so the way that we structured the lectures is one of us is the lecturer and one of us is basically the student and so the he was lecturing on the lecturer prepares all the materials comes up with the quizzes and then the student comes in not knowing anything so it's you know just like being on campus yeah uh and i was doing game theory in particular the prisoner's delivery's dilemma and so he needed to set up a little prisoner's dilemma grid so he drew it and i could see what he was drawing and the the prisoner's dilemma consists of two players two parties so he decided he would make little cartoons of the two of us and so there was uh two criminals right that were deciding whether or not to rat each other out um one of them he drew as you know a circle with a smiley face and a kind of goatee thing smooth head and the other one with all sorts of curly hair and he said this is smooth and curly i said smooth and curly he said no smoove with a v it's very important that it happened v and then talk actually and the students really the students really took to that like they've really they found that relatable he started singing smooth criminal by michael jackson yeah yeah yeah and that's those those names stuck so that so we now have a video series the an episode our kind of first actual episode should be coming out today um smoove and curly on video where the two of us discuss uh west episodes of westworld we watch westworld and we're like huh what does this say about computer science and ai and we've never we did not watch it i mean i know it's on season three or whatever we have as of this recording it's on season three and uh that's now two episodes total yeah i think it was three what do you think about westworld two episodes in so i can tell you guys so far yeah i'm just guessing what's going to happen next it seems like bad things are going to happen with the robot's uprisings so i no i have not i mean you know i vaguely remember a movie existing so i assume it's it's related to that but that was more my time than your time charles that's right because you're much older than i i think the important thing here is that uh it's narrative right it's all about telling a story that's the whole driving thing but the idea that they would give these reveries that they would make people they would make them remember remember the awful things that happened that happened who could possibly think that was good i got a i mean i don't know i've only seen the first two episodes or maybe the third one i think i've only said you know what it was you know what the problem is that the robots were actually designed by hannibal lecter that's true they were so like what do you think is going to happen it's a bad thing it's clear that things are happening and characters are being introduced and we don't yet know anything but still i was just struck by how it's all driven by narrative and story and there's all these implied things like programming hap the programming interface is talking to them about what's going on in their heads which is both i mean artistically it's probably useful to film it that way but think about how it would work in real life that just seems very crazy but there was we saw in the second episode there's a screen you could see things they were wearing like in the world it was quite interesting to just kind of ask this question so far i mean i assume it veers often to never neverland at some point but uh we can't answer that question i'm also a fan of a guy named alex garland he's a director of ex machina and he is the first i wonder if kubrick was like this actually is he like studies what would it take to program in ai systems like he's he's curious enough to go into that direction on the west wall side i felt there was more emphasis on the narratives than like actually asking like computer science questions yeah like how would you build this how would you uh and how would you debug it i still think to me that's the key issue they were terrible debuggers yeah and well they said specifically so we make a change and we put it out in the world and that's bad because something terrible could happen like if you're putting things out in the world and you're not sure whether something terrible is going to happen you're probably your process is probably i just feel like there should have been someone whose sole job it was to walk around and poke his head and say what could possibly go wrong just over and over again i would have loved if there was an and i did watch a lot more i'm not giving anything away i would have loved it if there was like an episode where like like the new intern is like debugging a new model or something and like it just keeps failing and they're like all right and then it's more turns into like a episode of silicon valley or something like that yes versus like all this ominous ai systems that are constantly like threatening the the fabric of this world that's been created yeah yeah and you know the other the this this reminds me of something that so i agree with that that actually be very cool at least well for the small percentage of people who care about debugging systems but the other thing is debugging the series it falls into think of the sequels fear of the debate oh my gosh and anyway so a nightmare show it's a it's a horror movie i think that's where we lose people by the way early on is the people who either decide either figure out debugging or think debugging is terrible this is where we lose people in computer science this is part of the struggle versus suffering right you you get through it and you kind of get the skills of it or you're just like this is dumb and this is a dumb way to do anything and i think that's when we lose people but um i well i'll leave it at that but i think that i think that that there's something really really neat about framing it that way but what i don't like about all of these pro all of these things and i love text mocking and by the way i thought the ending was very depressing um but again one of the things after talking to alex about he says that the thing that nobody noticed he put in is uh the at the end spoiler alert the the robot turns and looks at the camera and smiles right briefly and to him he thought that his definition of passing the touring the general version of the turing test or the consciousness test is smiling for no one hmm oh like like not oh you know it's it's like the chinese room kind of experiment it's not always trying to act for others right but just on your own being able to have a relationship with the actual experience and just like take it in i don't know he said like nobody noticed i mean the magic of it i had this vague feeling that i remember the smile but now you now you just put the memory in my head so probably not but i do think that that's interesting although by looking at the camera you are smiling for the audience right you're breaking the fourth wall it seems i mean well that's a that's a limitation in the medium but i i like that idea but here's the problem i have with all of those movies all of them um is that but i know why it's this way and i enjoy those movies um and westworld is uh it sets up the problem of ai as succeeding and then having something we cannot control but it's that's not the bad part of ai the bad part of ai is the stuff we're living through now right it's the using the data to make decisions that are terrible it's not the intelligence that's going to go out there and surpass us and you know take over the world or you know lock us into a room to starve to death slowly over multiple days it's instead uh the the tools that we're building that are allowing us to make the terrible decisions we would have less efficiently made before right you know computers are very good at making us more efficient including being more efficient at doing terrible things and and that's the part of the ai we have to worry about it's not the you know true intelligence that we're going to build sometime in the future probably long after we're around um but you know i i i i just i think that whole framing of it sort of misses the point even though it is inspiring and i was inspired by those ideas right that i got into this in part because i wanted to build something like that philosophical questions were interesting me but but you know that's not where the terror comes from the terror comes from the every day and you can construct situations in the subtlety of the interaction between ai and the human like with the with social networks all the stuff you're doing with uh interactive artificial intelligence but you know i i feel like cal 9000 came a little bit closer to that when it's in 2001 space odyssey because it felt like uh a personal assistant you know it felt like closer to the ai systems we have today and and the real things we might actually encounter which is over relying uh on in some fundamental way on our like dumb assistance or on social networks like over offloading too much of us onto uh you know onto things that require internet and power and so on and thereby becoming powerless as a stand-alone entity and then when that thing starts to misbehave in some subtle way it creates a lot of problems and those problems are dramatized when you're in space because you don't have a way to walk away well as the man said um once you once we started making the decisions for you it stopped being your world right that's the matrix michael in case you don't i didn't generally i don't remember but on the other hand i could say no because isn't that what we do with people anyway you know this kind of the shared intelligence that is humanity is relying on other people constantly to i mean we hyper-specialize right as individuals we're still generally intelligent we make our own decisions in a lot of ways but we leave most of this up to other people and that's perfectly fine and by the way everyone doesn't necessarily share our goals sometimes they seem to be quite against us sometimes we make decisions that others would see as against our own interests and yet we somehow manage it manage to survive i'm not entirely sure why an ai would actually make that worse or even different really you mentioned the matrix do you think we're living in a simulation it does feel like a thought game more than a real scientific question well i'll tell you why like i think it's an interesting thought experiment see what you think from a computer science perspective it's a good experiment of how difficult would it be to create a sufficiently realistic world that us humans would enjoy being in it that that's almost like if we're living in a simulation then i don't believe that we were put in the simulation i believe that it's just physics playing out and we came out of that like i don't i don't i don't think so you think you have to build the universe yeah i think the universe itself we can think of that as a simulation and in fact what i try sometimes i try to think about to understand what it's like for a computer to start to think about the world i try to think about the world things like quantum mechanics where it doesn't feel very natural to me at all and it really strikes me as i don't understand this thing that we're living in it has there's weird things happening in it that don't feel natural to me at all now if you want to call that as the result of a simulator okay i'm fine with that but like i don't know the bugs in the simulation there's the bugs i mean the interesting thing about the simulation is that it might have bugs i mean that that's the thing that i the but there would be bugs for the people in the simulation they're just that's just reality unless you were fair enough to know that there was a bug but i i think back to the matrix yeah the way you put this i don't think that we live in a in a simulation created for us okay i would say that i think that's interesting i've actually never thought about it that way i mean you the way you asked the question though is could you create a world that is enough for us humans it's an interestingly sort of self-referential question because the beings that created the simulation probably have not created a simulation that's realistic for them but we're in the simulation and so it's realistic for us so we could create a simulation that is fine for the people in the simulation as it were that would not necessarily be fine for us as the creators of the simulation but well you can you can forget i mean when you go into the if you play video games in virtual reality you can if with some suspension of disbelief or whatever yeah uh it becomes a world it becomes the world even like in brief moments you forget that another world exists i mean that's what like good stories do they pull you in and the question is is it possible to pull you know our brains are limited is it possible to pull the brain in to where we actually stay in that world longer and longer longer and longer and like not only that but we don't want to leave and so especially this is the key thing about the developing brain is if we journey into that world early on in life often how would you even know yeah yeah so i but like from a video game design perspective from a west world perspective it's i think i think it's an important thing for even uh computer scientists to think about because it's clear that video games are getting much better and virtual reality although it's been ups and downs just like artificial intelligence it feels like virtual reality will be here in a very impressive form if we were to fast forward 100 years into the future in a way that might change society fundamentally like if i were to i'm very limited in predicting the future as all of us are but if i were to try to predict like in which way i'd be surprised to see the world 100 years from now it'd be that or impressed it'd be that we're all no longer living in this physical world that we're all living in a virtual world you really need to be calculating god by sawyer it's a you'll read it in a night it's a very easy read but it's assuming you're that kind of reader but it's a it's a good story and it's kind of about this but not in a way that it appears and i uh really enjoyed the thought experiment um i think it's pretty sure it's robert sawyer but anyway he's he's apparently canadian's top science fiction writer which is why this story mostly takes place in toronto uh but it's a it's a very good uh it's a very good sort of story that sort of imagines this very different kind of simulation hypothesis sort of thing from say um the egg for example you know you know i'm talking about the short story um by the guy who did the martian who wrote the martian mm-hmm you know matt damon no the book so we had this whole discussion that michael doesn't uh doesn't partake in this exercise of reading yeah he doesn't seem to like it which seems very strange to me considering how much he has to read i read all the time i used to read 10 books at every week when i was a when i was in sixth grade or whatever i was a lot of it science fiction a lot of it a lot of history but i i love to read but anyway you should recalculating god i think you'll you'll it's very easy read like i said i think you'll enjoy sort of the ideas that it presents yeah i think the the thought experiment is quite interesting uh one thing i've noticed about people growing up now i mean we'll talk about social media but video games is a much bigger bigger and bigger and bigger part of their lives and then the video games have become much more realistic i think it's possible that the three of us are not uh and maybe the two of you are not familiar exactly with the numbers we're talking about here the number of people it's bigger than movies right it's it's it's huge i used to do a lot of the narrative computational narrative stuff i understand that economists can actually see the impact of video games on the labor market that there are there there's fewer young men of a certain age participating in like paying jobs than you'd expect and and that they trace it back to video games i mean the problem with star trek was not warp drive or teleportation it was the holodeck like if you have the holodeck that's it that's it you go in the holodeck you never come out i mean it just never made once i saw that i thought okay well so this is the end of humanity as we know it right they've invented the holiday because that feels like the singularity not some agi or whatever it's some possibility to go into another world that can be artificially made better than this one and slowing it down so you live forever or speeding it up so you appear to live forever or making the decision of when to die and then most of us will just be old people on the porch yelling at the kids these days in their virtual reality [Music] but they won't hear us because they've got headphones on so i mean rewinding back to moocs is there lessons that you've uh speaking of kids these days uh that was a transition all right i'll edit i'll fix it and post yeah that's charles's favorite phrase fix it in post fix it in post exit in post they said all when we were recording all the time whenever the editor didn't like something or whatever i would say we'll fix it in post he hated that yeah he hated that more than anything because charles's way of saying i'm not going to do it again [Laughter] you know you're on your own for this one but it always got fixed in post exactly right so uh is there something you've learned about i mean it's interesting to talk about moocs is there something you've learned about the process of education about thinking about the present i think there's two lines of conversation to be had here is the future of education in general that you've learned about and more presciently is the education in the times of covid yeah well the second thing in some ways matters more than the first um for at least in my head for the not just because it's happening now but because um i think it's it's reminded us of a lot of things coincidentally today there's an article out by a good friend of mine um who's also a professor of georgia tech but more importantly a writer and editor at the atlantic i named ian bogus um and the title is something like americans will sacrifice anything for the college experience and it's about why we went back to college and why people wanted us to go back to college and it's not you know greedy presidents trying to get the last dollar from someone it's because they want to go to college and what they're paying for is not the classes what they're paying for is the college experience it's not the education it's being there i've believed this for a long time that we continually make this mistake of people want to go back to college as being people want to go back to class they don't they want to go back to campus they want to move away from home they want to do all those things that people experience it's a rite of passage it's a it's a identity if i can if i can steal some of them ian's words here and i think that's right and i think what we've learned through kovid is it has made it the disaggregation was not the disaggregation of the education from the place universe the university place and that you can get the best anywhere you want to it turns out there's lots of reasons why that is not necessarily true the disaggregation is having it shoved in our faces that the reason to go again that the reason to go to college is not necessarily to learn it's to have the college experience and that's very difficult for us to accept even though we behaved that way most of us when we were undergrads you know a lot of us didn't go to every single class we learned and we got it and we look back on it and we're happy we had the learning experience as well obviously particularly us because this is the kind of thing that we do and my guess is that's true of the vast majority of your audience but that doesn't mean the i'm standing in front of you telling you this is the thing that people are excited about um and that's why they want to be there primarily why they want to be there so to me that's what coveted has forced us to deal with um even though i think we're still all in deep denial about it and hoping that it'll go back to that and i think about 85 of it will we'll be able to pretend that that's really the way it is again and we'll forget the lessons of this but technically what will come out of it or technologically will come out of it is a way of providing a more dispersed experience through online education and these kinds of remote things that we've learned and we'll have to come up with new ways to engage them in the experience of college which includes not just the parties or whatever kids do but the learning part of it so that they actually come out for five or six years later with having actually having actually learned something so um i think the world will be radically different afterwards and i think technology will matter for that just not in the way that the people who are building the technology originally imagined it would be and i think this would have been true even without covid but covet has accelerated that reality so it's happening in two or three years or five years as opposed to 10 or 15. that was an amazing answer that i did not understand so it was passionate and and i but i don't know i just didn't no i'm not trying to criticize it i think i'm i don't think i'm getting it so you mentioned disaggregation so what's that well so you know the power the power of technology that if you go on the west coast and hang out long enough is all about we're going to disaggregate these things together the books from the bookstore you know that kind of a thing and then suddenly amazon controls the universe right and technology is a disrupter right and people have been predicting that for uh higher education for a long time but certainly so is this is this the sort of idea like students can aggregate on a campus someplace and then take classes over the network anywhere yeah this is what people thought was going to happen or at least people claimed it was going to happen right that you know because my daughter is essentially doing that now she's on one campus but learning in a different campus sure and kobe makes that possible right um okovi makes that um league all but avoidable right but the idea originally was that you know you and i were going to create this machine learning class and it was going to be great and then no one else would be the machine learning class everyone takes right that was never going to happen but you know something like that but i feel like you didn't address that so why why why is it that why cue why i don't think that will be the thing that happens the college experience maybe i maybe i missed what the college experience was i thought it was peers like people hanging around a large part of it is peers well it's peers and independence yeah but none of that you can do classes online for all of that no no no no because no definitely we're social people right so you want to be able to take the classes that also has to be part of an experience it's in a context in the context of the university and by the way it actually matters that georgia tech really is different from brown i see because then students can choose the kind of experience they think is going to be best for them okay i think we're giving too much agency to the students in making an informed decision okay but the truth but yes they will make choices and they will have different experiences and some of those choices will be made for them some of them will be choices they're making because they think it's this that or the other i just don't want to say i don't want to give the idea not homogenous yes it's certainly not homogeneous right i mean georgia tech is different from brown brown is different from pick your favorite state school in iowa iowa state okay which i guess is my favorite state school in iowa sure but you know these are all different they have different contexts and a lot of those contexts are they're about history yes but they're also about the location of where you are uh they're about the larger group of people who are around you whether you're in athens georgia and you're basically the only thing that's there as a university you're responsible for all the jobs or whether you're at georgia state university which is an urban campus where you're surrounded by you know six million people uh in your campus where it ends and begins in the city ends it begins we we don't know it actually matters whether you're a small campus or a large campus why is it that if you go to georgia tech you're like forever proud of that and you like say that to people at dinner with like bars and whatever and if you not you know if you get a degree in an online university somewhere you don't that's not a thing that comes up at a bar well it's funny you say that so the students who take our online masters by several measures are more loyal than the students who come on campus certainly for the master's degree the reason for that i think and you'd have to ask them but based on my conversations with them i feel comfortable saying this is because this didn't exist before i mean we talk about this online masters and that it's reaching you know 11 000 students and that's an amazing thing and we're admitting everyone we believe we can succeed we got a 60 acceptance rate it's amazing right it's also a 6600 degree the entire degree costs 6 600 7 000 depending on how long you take a dollar degree as opposed to 46 000 cost you to come on campus so that feels and i can do it while i'm working full-time and i've got a family and a mortgage and all these other things so it's an opportunity to do something you wanted to do but you didn't think was possible without giving up two years of your life as well as all the money and everything else the life that you had built so i think we created something that's had an impact but importantly we gave a set of people opportunities they otherwise didn't feel they had so i think people feel very loyal about that my biggest piece of evidence for that besides the surveys is that we have somewhere north of 80 students might be 100 at this point who graduated but come back in ta for this class for basically minimum wage even though they're working full-time because they believe they believe in sort of having that opportunity and they want to be a part of something now will they will generation 3 feel this way 15 years from now will people have that same sense i don't know but right now they they kind of do and so it's not the online it's it's a matter of feeling as if you're a part of something right we're all very tribal yeah right um and i think there's something very tribal about being a part of something like that being on campus makes that easier going through a shared experience makes it easier it's harder to have that shared experience if you're alone looking at a computer screen we can create ways to make that is it possible it is that's the question is it still is the intuition to me and it was at the beginning when i saw something like the online master's program is that this is going to replace universities and it won't replace universities better but like where is it why because it's living in a different part of the ecosystem right the people who are taking it are already adults they've gone through their undergrad experience their i think their their goals have shifted from when they were 17. um they have other things that that are going right but it does do something really important something very social and very important right you know this whole thing about um you know don't build the sidewalks just leave the grass and the students or the people will walk and you put the sidewalks where they create paths kind of things yeah um their architects apparently believe that's the right way to do things the metaphor here is that we we created this environment we didn't quite know how to think about the social aspect but you know we didn't have time to solve all do all the social engineering right um the students did it themselves they created um you know these groups like on google plus they're like 30 something groups created in the first year because somebody had these google plus um and they created these groups and they divided up in ways that made sense we live in the same state or we're working on the same things we have the same background or whatever and they created these social things we sent them t-shirts and they were we have all these great pictures of students putting on their t-shirts as they travel around the world i climb to this mountaintop i'm putting this t-shirt on i'm a part of this they were they were part of them they created the social environment on top of the social network and the social media that existed uh to create this sense of belonging and being a part of something they found a way to do it right and i think they had other it scratched an itch that they had but they had scratched some of that itch that might have required they'd be physically in the same place long before right so i think yes it's possible and it's more than possible it's necessary but i don't think it's going to replace the university as we know it the university as we know it will change but there's just a lot of power and the kind of rite of passage kind of going off to yourself now maybe there'll be some other rite of passage that will happen right that's the best drive or somewhere else you can separate so the university is such a fascinating uh mess of things so just even the faculty position is a fascinating mess like it doesn't make any sense it it stabilizes itself but like why are the world-class researchers spending a huge amount of time of their time teaching and service like you're doing like three jobs yeah and and i mean it turns it's maybe an accident of history or human evolution i don't know it seems like the people who are really good at teaching are often really good at research there seems to be a parallel there but like it doesn't make any sense that you should be doing that at the same time it also doesn't seem to make sense that your place where you party is the same place where you go to learn calculus or whatever the but it's a safe space safe space for everything yeah relatively speaking it's a safe space now by the way i feel the need very strongly to point out that we are living in a very particular weird bubble right most people don't go to college and by the way the ones who do go to college they're not 18 years old right they're like 25 or something i forget the numbers you know the places where we've been where we are uh they look like whatever we think the traditional movie version of universities are but for most people it's not that way at all by the way most people who drop out of college it's entirely for financial reasons right the the so you know we were talking about a particular experience um and so for that set of people which is very small but larger than it was a decade or two or three or four certainly ago i don't think that will change my concern which i think is kind of implicit in some of these questions is that somehow we will divide the world up further uh into the people who get to have this experience and get to have the network and they sort of benefit from it and everyone else while increasingly requiring that they have more and more credentials in order to get a job as a barista right you got to have a master's degree in order to to work at starbucks i mean we're going to force people to do these things but they're not going to get to have that experience and there'll be a small group of people who do who continue to you know positive feedback et cetera et cetera i worry a lot about that which is why for me um and by the way here's an answer to your question about faculty which is why to me that you have to focus on access and the mission i think the reason whether it's good bad or strong i mean i agree it's strange but i think it's useful to have the faculty member particularly large r1 universities where we've all had experiences uh that you tie what they get to do and with the fundamental mission of the university and let the mission drive what i hear when i talk to faculty is they love their phd students because they're creating they're reproducing basically right and it lets them do their research and multiply but they understand that the mission is the undergrads and so they will do it without complaint mostly because it's a part of the mission and why they're here and they have experiences with it themselves and that it was important to get them get them where they were going the people tend to get squeezed in that by the way are the master students right who are neither the phds who are like us nor the undergrads we we have already bought into the idea that we we have to teach though that's increasingly changing anyway i think tying that mission in really matters and it gives you a way to unify people around making it an actual higher calling education feels like more of a higher calling to me than than even research because education you cannot treat it as a hobby if you're going to do it well but but that's the that's the pushback on this whole system is that you should education be a full-time job right and like it's almost like research is a distraction from that yes although i think most of our colleagues many of our colleagues would say that research is a job and education is the distraction right but that's the beautiful dance it seems to be that that tension in itself is seems to work seems to bring out the best in uh in the faculty or like that but i will point out two things one thing i'm going to point out the other thing i want michael to point out because i think michael is much closer to the to the to sort of the the ideal professor in some sense than i am well you're the platonic sense of a performance i don't know what he meant by that but he's he is a dean so he has a different experience i'm giving him i'm giving him time to think of the profound thing he's going to that's good but let me let me point this out which is that we have lecturers in the college of computing where i am uh there's 10 or 12 of them depending on your account as opposed to the 90 or so tenure track faculty those 10 lecturers who only teach well they don't only teach they also do service they some of them do research as well but primarily they teach they teach 50 over 50 of our credit hours and we teach everybody right so they're doing not just they're doing more than eight times the work of the tenure track faculty uh by just if more closer to nine or ten and that's including our grad courses right so they're doing this they're teaching more they're touching more more than anyone and they're beloved for it i mean so we recently had a survey we do these alumni everyone does these alumni surveys you hire someone from the outside to do whatever and and i was really struck by something you saw these really cool numbers i'm not going to talk about it because you know it's all internal confidential stuff but one thing i will talk about is there was a single question we asked our alum and these are people who graduated you know born in the 30s and 40s all the way up to people who graduated last week right um well that's great okay good um and here's the question name this a single person who had a strong positive impact on you something like that i think it was special impact yeah special impact on you and then so they got all the answers from people and they created a word cloud there was clear word cop created by people who don't do word clouds for a living because they had one person whose name like appeared nine different times like philip phil dr phil you know but whatever but they got all this and i looked at it and i noticed something really cool the five people from the college of computing i recognized were in that cloud and um four of them were lecturers the people who teach two of them relatively modern both were chairs of our division of computing instruction one just one retired one is going to retire soon and the other two were lecturers i remembered from the 1980s um two of those four by the way the fifth person was charles that's not important the thing is i i don't tell people that but the two of those people are teaching the words are named after thank you michael two of those are our teaching awards are named after right so when you ask students alumni people who are now 60 70 years old even you know who touch them they say the dean of students they say the big teachers who taught the big introductory classes that got me into it there's a guy named richard bark who's on there who's who's you know i who's known as a great teacher uh the the phil adler guy who um who uh i probably just said his last name wrong but i know the first name is phil because he kept showing up over and over again uh famous adler is what it said okay good but different people spelled it differently so he appeared multiple times right so he was a uh clearly he was a professor in um the business school um but when you read about himself i went to read a box i was curious who he was you know it's all about his teaching and the students that he touched right so whatever it is that we're doing and we think we're doing that's important or why we think the universities function the people who go through it yeah they remember the people who are kind to them the people who taught them something and they do remember it they remember it later i think that's important that's where the mission matters yeah not to completely lose track of the fundamental problem of how do we replace uh the the party aspect of universities uh before we go to the what makes the platonic professor do you uh do you think like what in your sense is the role of moocs in this whole picture during covid like are we should we desperately be clamoring to get back on campus or is this a stable place to be for a little while i don't know i know that it's that it's the online teaching experience and learning experience has been really rough i think that that people find it to be a struggle in a way that's not a happy positive struggle that when you got through it you just feel like glad that it's over as opposed to i've achieved something so you know i worry about that but um you know i worry about just even before this happened i worry about lecture teaching is how how well is that actually really working as far as a way to do education as a way to inspire people i mean all the data that i'm aware of seems to indicate and this kind of fits i think with charles's story is that people respond to connection right they actually feel if they're if they feel connected to the person teaching the class they're more likely to go along with it they're more they're more able to retain information they're more motivated to be involved in the class in some way and and that really matters it people can mean to the human themselves yeah so can't you do that actually perhaps more effectively online like you mentioned science communication so i i literally i think learned linear algebra from gilbert strang by watching mit open courseware when i was in drugs like and he was a personality he was a bit like a you know tiny in his tiny little world of math there's a bit of a rock star right so you kind of look up to that uh to that person can't that replace the in-person education it can help i will point out something i can't share the numbers but the we have surveyed our students and even though they have feelings about what i would interpret as connection i like that word um in the different modes of classrooms there's no difference between how they how well they think they're learning for them the thing that makes them unhappy is the situation they're in and i think the last lack of connection it's not whether they're learning anything they seem to think they're learning something anyway right in fact they seem to think they're learning it equally well uh presumably because the faculty are putting in or the instructors more generally speaking are putting in uh the energy and effort to try to make certain that they're what they've curated can be expressed to them in a useful way but the connection is missing and so there's huge differences in what they prefer and as far as i can tell what they prefer is more connection not less that connection just doesn't have to be physically in a classroom i mean look you know i used to teach 348 students in a machine learning class on campus you know why that was the biggest classroom on campus they're sitting in a theater they're sitting in theater seats i'm literally on a stage looking down on them and talking to them right there's no i mean we're not sitting down having a one-on-one conversation reading each other's body language trying to communicate and going we're not doing any of that so you know if you're on the if you're past the third row it might as well be online anyway is the kind of thing that people said daphne has actually said some version of this um that online starts on the third row or something like that and i think that's that's not yeah i like it i think it captures something important but people still came by the way they even the people who had access to our material would still come to class i mean there's a certain element about looking to the person next to you yeah it's just like their presence there their their boredom and like when the parts are boring and their excitement when the parts are exciting like and sharing in that like unspoken kind of uh yeah communication like in part the connection is with the other people in the room watching watching the circus on tv alone is ever been to a movie theater and been the only one there at a comedy it's not as funny as when you're in a room full of people all laughing well you need maybe you need just another person it's like as opposed to many maybe maybe there's some kind of well there's different kinds of connection right and there's different kinds of comedy well in the sense as we're learning today i wasn't sure if that was going to land but um just the idea that that different jokes i i've i've now done a little bit of stand up and so different jokes work in different size crowds too right where sometimes if you know if it's a big enough crowd then even a really subtle joke can take root someplace and then that cues other people and it kind of there's a whole statistics of i did this terrible thing to my brother so when i was really young i decided that my brother was only laughing at sitcoms when i laughed like he was taking cues from me so i like purposely didn't laugh just to see if i was right laugh at non-funny things yes i really want to do both sides i did both sides and uh and at the end of it i told him what i did he was very upset about this yeah and from that day on he's he lost his sense of humor no no no no well yes but from that day on he he he laughed on his own he stopped taking cues from me so i want to say that you know it was a good thing that i did but yes yes you was making that man's life yes but it was mostly me but it's true though it's true right that people i i think you're right but okay so that's where does that get us that gets us the idea that i mean certainly movie theaters are a thing right where people like to be watching together even though the people on the screen can't aren't really co-present with the people in the audience the audience is co-present with themselves by the way and that point it's it's an open question that's being raised by this whether movies will no longer be a thing because netflix's the audience is growing so that's it's a it's a very parallel question for education will move and theaters still be a thing right in 2000 but i think i think the argument is that there is a feeling of being in the crowd that isn't replicated by being at home watching it and that there's value in that and then i think just but it scales better on right i feel like we're having a conversation about whether concerts will still exist after the invention of the record or the cd or wherever it is right you're right concerts are dead um well okay i think the joke is only funny if you say it before now right yeah like three years ago it's like well no obviously i'll wait to publish this until we have a vaccine you know we'll fix it in post but i think the the important thing is the virus bust concerts changed right first of all movie theaters weren't this way right in like the 60s and 70s they weren't like this like blockbusters were basically what jaws and star wars created blockbusters right before then there weren't like the whole summer shared summer experience didn't exist in our lifetimes right certainly you were well into adulthood by the time this was true right so it's just a very different it was it's very different so what the what we're we've been experiencing in the last 10 years is not like the majority of human history but more importantly concerts right concerts mean something different most people don't go to concerts anymore like there's an age where you care about it you sort of stop doing it you keep listening to music or whatever so i think that's a painful way of saying that um it will change it's not the same things are going away replace is too strong of a word but it will change it has to i actually like to push back i wonder because i think you're probably just throwing that your intuition out oh i won't wait and turn it's possible that concerts more people go to concerts now but obviously much more people listen to well just dumb when they then before there was records uh it's it's possible to argue that uh if you look at the data uh that it just expanded the pie of what music listening means so it's possible that like universities grow in the parallel where the theaters grow but also more people get to watch movies more people get to like be educated yes i i hope that yeah and to this extent that we can grow the pie and have education be not just something you do for four years when you're done with your other education but it'd be a more lifelong thing that would have tremendous benefits especially as the the economy and the world change rapidly like people need opportunities to stay abreast of these changes and so i don't know i could i could it's all part of the the ecosystem it's all to the good i mean you know i'm not gonna have an argument about whether we we lost fidelity when we went from laserdisc to dvds or record players to cds i mean i'm i'm willing to grant that that is true but convenience matters and the ability to do something that you couldn't do otherwise because that convenience matters and you can tell me i'm only getting 90 of the experience but i'm getting the experience i wasn't getting it before or wasn't lasting as long or it wasn't i mean this just seems this just seems straightforward to me it's gonna it's going to change it is for the good that more people get access and it is our job to do two separate things one to educate them and make access available that's our mission but also for very simple selfish reasons we need to figure out how to do it better so that we individually stay in business we can do both of those things at the same time they are not in they may be intentioned but they are not mutually exclusive so you've educated some scary number of people so you've seen a lot of people succeed find their path to life is there advice that you can give to a young person today about computer science education about education in general about life about uh whatever the journey that one takes in there maybe in their teens in their early twenties instead of in those underground years as you try to go through the essential process of partying and not going to classes and yet somehow trying to get a degree if you get to the point where you're you're you're far enough up in the in the hierarchy of needs that you can actually make decisions like this then find the thing that you're passionate about and pursue it and sometimes it's the thing that drives your life and sometimes it's secondary and you'll do other things because you've got to eat right you got a family you got to feed you've got people you have to help or whatever i and i understand that and it's not easy for everyone but um always take a moment or two to pursue the things that you love the things that bring passion and happiness to your life and if you don't i know that sounds corny but i genuinely believe it and if you don't have such a thing then you're lying to yourself you have such a thing you just have to find it and it's okay if it takes you a long time to get there rodney dangerfield became a comedian in his 50s i think certainly wasn't his 20s and lots of people failed for a very long time before getting to where they were going um you know i try to have hope and it it wasn't obvious i mean you know we you and i talked about the experience that i had um a long time ago with with a particular police officer was it my first one it wasn't my last one um but you know in my view i wasn't supposed to be here after that and i'm here so it's all gravy so you might as well go ahead and grab life as you can because of that that's that's sort of how i see it while recognizing again the delusion matters right allow yourself to be deluded allow yourself to believe that it's all going to work out just don't be so deluded that you you miss the obvious and and you're going to be fine it's going to be there it's going to be there it's going to work out what do you think i like to say choose your parents wisely because that has a big impact on your life yeah i mean you know i mean there's a whole lot of things that you don't get to pick um and and and whether you get to have you know one kind of life or a different kind of life can depend a lot on things out of your control but i really do believe in the in the passion excitement thing my i was talking to my mom on the phone the other day and essentially what came out is that computer science is really popular right now and and and i get to be a professor teaching something that's very uh attractive to people and she's she was like trying to give me some appreciation for how forsightful i was for choosing this line of work as if somehow i knew that this is what was going to happen in 2020 uh but that's not how it went for me at all like i studied computer science because i was just interested it was just so interesting to me i didn't i didn't think it would be particularly lucrative yeah and i've done everything i can to keep to keep it as unlucrative as possible yeah um some of my you know some of my friends and colleagues have have have not done that and i pride myself on my ability to just to remain unrich but um but but i think but but i do believe that that like i'm glad i mean i'm glad that it worked out for me it could have been like oh what i was really fascinated by is this particular kind of engraving that nobody cares about but um so i got lucky and the thing that i cared about happened to be a thing that other people eventually cared about but i don't think i would have had a fun time choosing anything else like this was the thing that kept me interested and engaged well one thing that people tell me especially around as an early undergraduate and the internet is part of the problem here is they say they're passionate about so many things how do i choose a thing which is a harder thing for me to know what to do with is there any i mean don't you know i mean you know a long time ago i walked down the hallway and i took a left turn yeah i could have taken a right turn and my world could be better or it could be worse i have no idea i have no way of knowing is there anything about this particular hallway that's relevant or you're just in general choices yeah you were on the left it sounds like you regret not taking the right oh no not at all you brought it up well because there was a turn there on the left was michael lemon's office right i mean these sorts of things happen right yes but here's the right by the way it was just a blank wall it wasn't a huge choice it would have really hurt he tried first no but it's it's true right that you know i i think about ron brockman right i i went i took a trip i wasn't supposed to take and i ended up talking to to um on about this and i ended up going down this entire path uh that allowed me to i think get tenure but by the way i decided to say yes to something that didn't make any sense and i went down this educational path but it would have been you know who knows right maybe if i hadn't done that i would be a billionaire right now i'd be elon musk my life could be so much better my life could also be so much worse you know you just got to feel that sometimes you have decisions you're going to make you cannot know what's going to you should think about it right some things are clearly smarter than other things you've got to play the odds a little bit but in the end if you've got multiple choices or lots of things you think you might love go with the thing that you actually love the thing that jumps out at you and sort of pursue it for a little while the worst thing that will happen is you took a left turn instead of a right turn and you ended up merely happy beautiful so so accepting so taking the step and just accepting accepting that that don't like question questions life is long and there's time to actually pursue every once in a while uh you have to put on a leather suit and make a thriller video every once in a while yeah uh i was told that you actually danced but that part was edited out i don't dance there was a thing where we did do the uh yeah the zombie thing yeah we did do the zombies yeah but that wasn't edited out it just wasn't able to put into the final thing i'm quite happy but there was a reason for that too right like i wasn't wearing something right there was a reason for that i can't remember what it was no i love this suit is that what it was i can't remember anyway the right thing happened exactly you took the left turn and then the third of the right ended up being the right thing so a lot of people ask me that are a little bit tangential to the programming the computing world and they're interested to learn programming like all kinds of disciplines that are outside of the particular discipline of computer science what advice do you have for people that want to learn how to program or want to either taste this little skill set or discipline or try to see if it can be used somehow in their own life what stage of life are they in uh it feels one of the magic things about the internet of the people that write me is i don't know because my answer is different for for my daughter is taking ap computer science right now hi johnny um she's uh she's amazing and doing amazing things and my son's beginning to get interested and i'll be really curious where he takes it i think he's his mind actually works very well for this sort of thing and she's doing great but one of the things i have to tell her all the time she points well i want to make a rhythm game so i want to go for two weeks and then build a rhythm game show me how to build a rhythm game and start small learn the building blocks and hours take the time have patience eventually you'll build a rhythm game i was in grad school when i suddenly woke up one day over the royal east um and i thought wait a minute i'm a computer scientist i should be able to write pac-man in an afternoon and i did not with great graphics it was actually a very cool game i had to figure out how the ghost moved and everything and i did it in an afternoon in pascal on an old apple 2gs um but if i had started out trying to build pac-man i think it probably would have ended very poorly for me luckily back then there weren't you know these magical devices we call phones and software everywhere to give me this illusion that i could create something by myself from the basics inside of a weekend like that i mean that was a culmination of years and years and years right before i decided i should be able to write this and i could so you know my advice if you're early on is you know you've got the internet there are lots of people there to give you the information find someone who cares about this remember they've been doing it for a very long time take it slow learn the little pieces get excited about it and then keep the big projects you want to build in mind you'll get there soon enough because as a wise man once said life is long sometimes it doesn't seem that long but it is long and you'll have enough time to to build it all out it all the information is out there but start small you know generate different object numbers that's not exciting but it'll program well there's only one programming language it's lisp but if you have to pick a programming language i guess in today's what would i do i guess i do python is basically doing this but with better syntax blasphemy yeah see with c syntax how about that so you're going to argue that c syntax is better than anything anyway also i'll go i'm going to answer python despite tell me tell your story about the somebody's dissertation that had a lisp program in it it was so funny this is a this is dave's dave's listening to him he was like dave mcallister who was a professor at mit for a while and and then he came in our in our girl labs now he's at um now he's a technology technical institute of chicago uh a brilliant guy uh such an interesting guy anyway his thesis uh it was a theorem proverb and he decided to have as an appendix uh his actual code which of course was in list because of course it was it's like the last 20 pages are just right parentheses it's just wonderful it's like they that's programming right there just like pages of pages of right parentheses anyway lisp is the only real language but i understand that that's not necessarily the place where you start python is just fine nah python is good if you're you know of a certain age if you're really young and trying to figure it out graphical languages that let you kind of see how the thing works and that's fine too they're all fine it almost doesn't matter but there are people who spend a lot of time thinking about how to build languages that get people in the questions are you trying to get in and figure out what it is or do you already know what you want and that's why i asked you what stage of life people are in because if you're different stages of life you you would you would attack it differently the the answer to that question of which language keeps changing i mean there's some value to exploring uh a lot of people write to me about julia there's there's these like more modern languages they keep being invented rust and and kotlin and there's stuff that uh for people who love functional languages like lisp there apparently there's echoes of that but much better in the modern languages and it's worthwhile to uh especially when you're learning languages it feels like it's okay to try one that's not like the popular one oh yeah but you know you i think you get that you get that way of thinking almost no matter what language and if you if you push far enough like it can be assembly language but you need to push pretty far before you start to hit the really deep concepts that you would get sooner in other languages but like i don't know computation is kind of computation it's kind of touring equivalent is kind of computation and so it's so it matters how you express things but you have to build out that mental structure in your mind and you i don't i don't think it super matters which language i mean it matters a little because some things are just at the wrong level of abstraction i think assembly's at the wrong level abstraction for someone coming in new um i think that if you start someone coming in new yes for frameworks big frameworks or or quite a bit um you got to get to the point where i want to learn any language means i just pick up a reference book and i think of a project and i go through it in the weekend right you got it you got to get there you're right though the languages that are designed for that are it almost doesn't matter pick the ones that people have built tutorials and infrastructure around to help you get kind of kind of eased into it because it's hard i mean we i did this little experiment once um i was teaching intro to cs in the summer as a favor uh which is anyway i was using memories i was teaching introduces a favor it was very funny because i'd go in every single time and i would think to myself how am i possibly going to fill up an hour and a half talking about for loops right and there wasn't enough time it took me a while to realize this right there are only three things right there's reading from a variable writing to a variable in conditional branching everything else is syntactic sugar right the syntactic sugar matters but that's it and when i say that's it i don't mean it's simple i mean it's hard like conditional branching loops variable those are really hard concepts so you shouldn't be discouraged by this here's a simple experiment i'm going to ask you a question now you ready x equals three okay y equals four okay what is x three what is y four y equals gonna mess this up no that's oh it's easier y equals x y equals x what is y uh three that's right x equals 7 what is y that's one of the trickiest things to get for programmers that there's a memory and the variables are pointing to a particular thing in memory and sometimes the languages hide that from you and they bring it closer to the way you think mathematics works right so in fact mark gosdale who worries about these sorts of things or used to worry about these sorts of things anyway had this kind of belief that actually people when they see these statements x equals something y equals something y equals x that you have now uh made a mathematical statement that y and x are the same which you can if you just put like an anchor in front of it yes but people that's not what you're doing yeah right i thought and i kind of asked the question and i i think had some evidence for this hardly a study is that most of the people who didn't know the answer weren't sure about the answer they had used spreadsheets ah and so it's a it's a na it's it's it's you know um it's by it's by reference or by name really right and so depending upon what you think they are you get completely different answers the fact that i could go or one could go two thirds of the way through a semester and people still hadn't figured out in their heads when you say y equals x what that meant tells you it's actually hard because all those answers are possible in fact when you said oh if you just put an ampersand in front of it i mean that doesn't make any sense for an intro class and of course a lot of language don't even give you the ability to think about it in terms of ampersand do we want to have a 45-minute discussion about the difference between equal eq and equal in lisp yeah i know you do [Laughter] but you know you could do that it's this is actually really hard stuff so you shouldn't be it's not too hard we all do it but you shouldn't be discouraged it's why you should start small so that you can figure out these things you have the right model in your head so that when you write the language you can execute it and build the machine that you you want to build right yeah the funny thing about programming on those very basic things is the the the very basics are not often made explicit which is actually what drives everybody away from basically any discipline but program is just another one like even a simpler version of the equal sign that i kind of forget is in mathematics equals is not assignment yeah right like i think basically every single programming language with just a few handful of exceptions equals his assignment and you have some other operator for uh equality yeah and you know even that like everyone kind of knows it once you started doing it but like you need to say that explicitly or you just realize it like yourself otherwise you'll be you might be stuck for you said like half a semester you could be stuck for quite a long time and i think also part of the programming is being okay in that state of confusion for a while it's it's to the debugging point it's like i just wrote two lines of code why doesn't this work and staring at that for like hours and trying to figure out and then every once in a while you just have to restart your computer and everything works again and then and then you just kind of stare into the void with the tear slowly rolling down your eye by the way the fact that they didn't get this actually had no impact on i mean they were still able to redo their assignments right because it turns out their misunderstanding wasn't being revealed to them yes by the problem sets we were found actually yeah i wrote a um a program a long time ago actually for my master's thesis and uh in c plus i think or c i guess we'll see and it was uh all memory management and terrible um and it wouldn't work for a while and it was some kind of it was clear to me that it was overwriting memory and i just couldn't i was like look i got a paper time for this so i basically declared a variable at the front in the main that was like 400k just an array and it worked because wherever i was scribbling over memory it would scribble into that space and it didn't matter and so i never figured out what the bug was but i did create something to sort of deal with it to work around it and it you know that's crazy that's crazy it was okay because that's what i wanted but i knew enough about memory managed to go you know management to go you know i'm just going to create an empty array here and hope that that deals with the scribbling memory problem and it did that takes a long time to figure out and by the way the language you first learned probably this garbage collection anyway so you're not even going to come up across it you're going to come across that problem so we talked about the the minsky idea of hating everything you do and hating yourself so let's end on a question that's gonna make both of you very uncomfortable okay which is what is your charles what's your favorite thing that you're grateful for about michael and michael what is your favorite thing that you're grateful for about charles well that answer is actually quite easy his friendship he's still the easy i did yeah i'll tell you what i hate about charles that steals my good answers the thing i like most about charles he sees the world in it in a similar enough but different way that i it's sort of like having another life it's sort of like i get to experience things that i wouldn't otherwise get to experience because i would not naturally gravitate to them that way and so he just he just shows me a whole other world it's awesome yeah the the inner product is not zero for sure it's not quite one point seven maybe just enough that you can learn just enough that you can learn that's the definition of friendship the inner product is 0.7 yeah i think so that's the answer to life really charles sometimes believes in me when i have not believed in me he can he also sometimes works as an outward confidence that he has so much so much confidence and self i don't know aware comfortableness okay let's go with that um that i feel better a little bit if he if he thinks i'm okay then maybe i'm not as bad as i think i am at the end of the day luck favors the charles it's a huge honor to talk with you thank you so much for taking this time wasting your time with me it was an awesome conversation you guys are an inspiration to a huge number of people and to me so really enjoyed this thanksgiving i enjoyed it as well thank you so much and by the way if luck favors the charleston is certainly the case that i've been very lucky to know i'm going to edit that part out thanks for listening to this conversation with charles isabel and michael littman and thank you to our sponsors athletic greens super nutritional drink ate sleep self-cooling mattress master class online courses from some of the most amazing humans in history and cash app the app i use to send money to friends please check out the sponsors in the description to get a discount and to support this podcast if you enjoy this thing subscribe on youtube review it with five stars napa podcast follow on spotify support it on patreon connect with me on twitter at lex friedman and now let me leave you some words from desmond tutu don't raise your voice improve your argument thank you for listening and hope to see you next time