Cristos Goodrow: YouTube Algorithm | Lex Fridman Podcast #68
the following is a conversation with Christos Kudrow vice president of engineering at Google and head of search and discovery at YouTube also known as the YouTube algorithm YouTube has approximately 1.9 billion users and every day people watch over 1 billion hours of YouTube video it is the second most popular search engine behind Google itself for many people it is not only a source of entertainment but also how we learn new ideas from math and physics videos to podcasts to debates opinions ideas from out-of-the-box thinkers and activists some of the most tense challenging and impactful topics in the world today YouTube and other content platforms receive criticism from both viewers and creators as they should because the engineering task before them is hard and they don't always succeed and the impact of their work is truly world-changing to me YouTube has been an incredible wellspring of knowledge I've watched hundreds if not thousands of lectures that changed the way I see many fun about those ideas in math science engineering and philosophy but it does put a mirror to ourselves and keeps the responsibility of the steps we take in each of our online educational journeys into the hands of each of us the YouTube algorithm has an important role in that journey of helping us find new exciting ideas to learn about that's a difficult and an exciting problem for an artificial intelligence system as I've said in lectures and other forums recommendation systems will be one of the most impactful areas of AI in the 21st century and YouTube is one of the biggest recommendation systems in the world this is the artificial intelligence podcast if you enjoy it subscribe on YouTube give it five stars an Apple podcast follow on Spotify supported on patreon or simply connect with me on Twitter Alex Friedman spelled Fri D ma a.m. I recently started doing ads at the end of the introduction I'll do one or two minutes after introducing the episode and never any ads in the middle that can break the flow of the conversation I hope that works for you and doesn't hurt the listening experience this show is presented by cash app the number one finance app in the App Store I personally use cash app to send money to friends but you can also use it to buy sell and deposit Bitcoin in just seconds cash app also has a new investing feature you can buy fractions of a stock say $1 worth no matter what the stock price is brokerage services are provided by cash app investing a subsidiary of square and member si PC I'm excited to be working with cash app to support one of my favorite organizations called first best known for their first robotics and Lego competitions they educate and inspire hundreds of thousands of students in over 110 countries and have a perfect rating and Charity Navigator which means that donated money is used to maximum effectiveness when you get cash app from the App Store Google Play and use code Lex podcast you'll get ten dollars in cash app will also donate ten dollars to the first which again is an organization that I've personally seen inspire girls and boys the dream of engineering a better world and now here's my conversation with Christos Gaudreau YouTube is the world's second most popular search engine behind Google of course we watch more than 1 billion hours of YouTube videos a day more than Netflix and facebook video combined YouTube creators upload over 500 thousand hours of video every day average lifespan of a human being just for comparison is about 700,000 hours so what's uploaded every single day is just enough for a human to watch in a lifetime so let me ask an absurd philosophical question if from birth when I was born and there's many people born today with the internet I watched YouTube videos non-stop do you think there are trajectories through YouTube video space that can maximize my average happiness or maybe education or my growth as a human being I think there are some great trajectories through YouTube videos but I wouldn't recommend that anyone and all of their waking hours or all of their hours watching YouTube I mean I think about the fact that YouTube has been really great for my kids for instance my oldest daughter you know she's been watching YouTube for several years she watches Tyler Oakley and the vlogbrothers and I know that it's had a very profound and positive impact on her character and my younger daughter she's a ballerina and her teachers tell her that YouTube is a huge advantage for her because she can practice a routine and watch like professional dancers do that same routine and stop it and back it up and rewind and all that stuff right so it's been really good for them and then even my son is a sophomore in college he he got through his linear algebra class because of a channel called three blue one brown which you know helps you understand linear algebra but in a way that would be very hard for anyone to do on a whiteboard or a chalkboard and so I think that those experiences from my point of view were very good and and so I can imagine really good trajectories through YouTube yes have you looked at do you think of broadly about that trajectory over a period cuz YouTube was growing up now so over a period of years you just kind of gave a few anecdotal examples but you know I used to watch certain shows on YouTube I don't anymore I've moved on to other shows and ultimately you want people to from YouTube's perspective to stay on YouTube to grow as human beings on YouTube so you have to think not just what makes them engage today or this month but also over a period of years absolutely that's right I mean if YouTube is going to continue to enrich people's lives then you know then it has to grow with them and and people's interests change over time and so I think we've we've been working on this problem and I'll just say it broadly is like how to introduce diversity and introduce people who are watching one thing to something else they might like we've been working on that problem all the eight years I've been at YouTube it's a hard problem because I mean of course it's trivial to introduce diversity that doesn't help a random video I could just randomly select a video from the billions that we have it's likely not to even be in your language so haha the likelihood that you would watch it and develop a new interest is very very low and so what you want to do when you're trying to increase diversity is find something that is not too similar to the things that you've watched but also something that you might be likely to watch and that balance finding that spot between those two things is quite challenging so the diversity of content diversity of ideas it's uh it's a really difficult it's the thing like that's almost impossible to define alright like what's different so how do you think about that so two examples is a I'm a huge fan of three blue one Brown say and then one diversity I wasn't even aware of a channel called veritasium width which is a great science physics whatever channel so one version of diversity is showing me Derek's veritasium channel which I was really excited to discover actually now watch a lot of his videos okay so you're a person who's watching some math channels and you might be interested in some other science or math channels so like you mentioned the first kind of diversity is just show you some some things from other channels that are related but not just you know not all the three blue one Brown Channel throw in a couple others so so that's the maybe the first kind of diversity that we started with many many years ago taking a bigger leap is is about I mean the mechanisms we do we use for that is is we basically cluster videos and channels together mostly videos we do every almost everything at the video level and so we'll we'll make some kind of a cluster some embedding process and then and then measure you know what is the likelihood that a that users who watch one cluster might also watch another cluster that's very distinct so we may come to find that that people who watch science videos also like jazz this is possible right and so and so because of that relationship that we've identified through the through the embeddings and then the measurement of the people who watch both we might recommend a jazz video once in a while so there's this clustering the embedding space of jazz videos and science videos and so you kind of try to look at aggregate statistics where if a lot of people that jump from science cluster to the jazz cluster tend to remain as engaged or become more engaged then that's that means those two are they should hop back and forth and they'll be happy right there's a higher likelihood that a person from who's watching science would like jazz then the person watching science would like I don't know backyard railroads or something else right and so we can try to measure these likelihoods and use that to make the best recommendation we can so okay so we'll talk about the machine learning of that but I have to linger on things that neither you or anyone have an answer to there's gray areas of truth which is for example now I can't believe I'm going there but politics it it happens so that the certain people believe certain things and they're very certain about them let's move outside the red versus blue politics of today's world but there's different ideologies for example in college I read quite a lot of iron rand i studied and that's a particular philosophical ideologies I find I found it interesting to explore okay so that was that kind of space I've kind of moved on from that cluster intellectually but it nevertheless is an interesting cluster there's I was born in Soviet Union socialism communism is a certain kind of political idea that's really interesting to explore again objectively just there's a set of beliefs about how the economy should work and so on and so it's hard to know what's true or not in terms of people within those communities they're often advocating that this is how we achieve utopia in this world and they're pretty certain about it so how do you try to manage politics in this chaotic divisive world not positively any kind of ideas in terms of filtering what people should watch next and in terms of also not letting certain things be on YouTube this is exceptionally difficult responsibility right well the responsibility to get this right is our top priority and and the first comes down to making sure that we have good clear rules of the road right like just because we have freedom of speech doesn't mean that you can literally say anything right like we as a society have accepted certain restrictions on our freedom of speech there are things like libel laws and things like that and so where we can draw a clear line we do and we continue to evolve that line over time however as you point it out wherever you draw the line there's going to be a border line and in that border line area we are going to maybe not remove videos but we will try to reduce the recommendations of them or the proliferation of them by demoting them and then alternatively in those situations try to raise what we would call Thorat ativ or credible sources of information so we're not trying to I mean you mentioned Iran and communism you know those are those are two like valid points of view that people are going to debate and discuss and and of course people who believe and one of the other of those things are going to try to persuade other people to their point of view and so we're not trying to settle that choose a side or anything like that what we're trying to do is make sure that the the people who are expressing those point of view and and offering those positions are authoritative and credible so let me ask a question about people I don't like personally you heard me I don't care if you leave comments on this is uh and but sometimes there's brilliantly funny which is trolls so it's people who kind of mock I mean the Internet is full the reddit of mock style comedy where people just kind of make fun of point out that the emperor has no clothes and there's brilliant comedy and that but sometimes it can get cruel and mean so on that on the mean point and sorry to linger on these things that have no good answers but it actually is I totally hear you that this is really important you're trying to solve it but how do you reduce the meanness of people on YouTube I understand that anyone who uploads YouTube videos has to become resilient to a certain amount of meanness like I've heard that from many creators and we would we are trying in various ways comment ranking allowing certain features to block people to reduce or or make that that meanness or that trolling behavior less effective on YouTube yeah and so I mean it's it's very important but it's something that we're we're gonna keep having to work on and and you know as we improve it maybe we'll get to a point where where people don't have to suffer this sort of meanness when they upload YouTube videos I hope we do but you know but it just does seem to be something that you have to be able to deal with as a YouTube creator now it is do you have a hope that he mentioned two things that can I agree was so there's a machine-learning approach of ranking comments based on whatever based on how much they contribute to the healthy conversation let's put it that way then the other is almost an interface question of how do you how does the Creator filter so block or how does how do humans themselves the users of YouTube manage their own conversation do you have hope that these two tools will create a better society without limiting freedom of speech too much without sort of Antonin even like saying that people what do you mean limiting sort of curating speech I mean I think that that overall is our whole project here at YouTube right like yeah we fundamentally believe and I personally believe very much that YouTube can be great it's been great for my kids I think it can be great for society but it's absolutely critical that we get this responsibility part right and that's why it's our top priority Susan Wojcicki who's the CEO of YouTube she says something that I personally find very inspiring which is that we want to do our jobs today in a manner so that people 20 and 30 years from now will look back and say you know YouTube they they really figured this out they really found a way to strike the right balance between the openness and the value that the openness has and also making sure that we are meeting our responsibilities to users in society so the burden on YouTube actually is quite incredible and the one thing that people don't I don't give enough credit to the seriousness and the magnitude of the problem I think so I I personally hope that you do solve it because a lot is in your hand a lot is riding on your success or failure so it's besides of course running a successful company you're also curating the content of the internet and the conversation the internet that's a powerful thing so one thing that people wander about is how much of it can be solved with pure machine learning so looking at the data studying the data and creating algorithms that curate the comments curate the content and how much of it needs human intervention meaning people here YouTube in a room sitting and thinking about what is the nature of truth what is what are the ideals that we should be promoting that kind of thing so algorithm versus human input what's your sense I mean my own experience has demonstrated that you need both of those things algorithms I mean you're familiar with machine learning algorithm and the thing they need most is data and the data is generated by humans and so for instance when we're building a system to try to figure out which are the videos that are misinformation or borderline policy violations well the first thing we need to do is get human beings to make decisions about which which of those videos are in which category and then we use that data and and basically you know take that information that's that's determined and governed by humans and and extrapolated or apply it to the entire set of billions of YouTube videos and we couldn't we we couldn't get to all the videos on YouTube well without the humans and we we couldn't use the humans to get to all the videos of YouTube so there's no world in which you have only one or the other of these things and just as you said a lot of it comes down to people at YouTube spending a lot of time trying to figure out what are the right policies you know what are the outcomes based on those policies are they the kinds of things we want to see and then once we kind of get a get an agreement or or build some consensus around around what the policies are well then we've got to find a way to implement those policies across all of YouTube and that's where both the human beings we call them evaluators or reviewers come into play to help us with that and then and then once we get a lot of training data from them then we apply the machine learning techniques to take it even further do you have a sense that these human beings have a bias in some kind of direction sort of I mean that's the interesting question we do sort of in autonomous vehicles and computer vision in general a lot of annotation and we rarely ask what bias do the annotators have you know the it even in the sense that they're better than they're better anting certain things and others for example people are much better at annotating segmentation at segmenting cars in a scene versus segmenting bushes or trees you know there's specific mechanical reasons for that but also because the cement its semantics gray area and and just for a lot of reasons people are just terrible at annotating trees okay so in the same kind of sense do you think of in terms of people reviewing videos or annotating the content of videos is there some kind of bias that you're aware of or seek out in that human input well we take steps to try to overcome these kinds of biases or biases that we think would be problematic so for instance like we asked people to have a bias towards scientific consensus that's something that we we instruct them to do we ask them to have a bias towards demonstration of expertise or credibility or authoritative nests but there are other biases that we that we want to make sure to try to remove and there's many techniques for doing this one of them is you send the same thing to be reviewed to many people and so you know that's one technique another is that you make sure that the people that are doing these sorts of tasks are from different backgrounds and different areas of the United States of the world but then even with all of that it's possible for certain kinds of what we would call unfair biases to creep into machine learning systems primarily as you said because maybe the training data itself comes in in in a biased way and so we also have worked very hard on the improving the machine learning systems to remove and reduce unfair biases when it's when it goes against or or has involved some protected class for instance thank you for exploring with me some of the more challenging things I'm sure there's a few more that we'll jump back to but let me jump into the fun part which is maybe the basics of the quote-unquote YouTube algorithm what is the YouTube algorithm look at to make recommendation for what to watch next was from a machine learning perspective or when you search for a particular term how does it know what to show you next because it seems to at least for me do an incredible job both well that's kind of you to say it didn't used to do a very good job but it's gotten better over the years even even I observed that it's improved quite a bit those are two different situations like when you search for something YouTube uses the best technology we can get from Google to make sure that that the YouTube search system finds what someone's looking for and of course the very first things that one thinks about is okay well does the word occur in the title for instance you know but there but there are much more sophisticated things where we're mostly trying to do some syntactic match or or maybe a semantic match based on words that we can add to the document itself for instance you know maybe is is this video watched a lot after this query right that's something that we can observe and then as a result make sure that that that document would be retrieved for that query now when you talk about what kind of videos would be recommended to watch next that's something again we've been working on for many years and probably the first the first real attempt to do that well was to use collaborative filtering so you can't describe what collaborative filtering is sure it's just basically what we do is we observe which videos get watched close together by the same person and if you observe that and if you can imagine creating a graph where the videos that get watched close together by the most people are sort of very close to one another in this graph and videos that don't frequently get watch close too close together by the same person or the same people are far apart then you end up with this graph that we call the related graph that basically represents videos that are very similar or related in some way and what's amazing about that is that it puts all the videos that are in the same language together for instance and we didn't even have to think about language just does it yeah I didn't it puts all the videos that are about sports together and it puts most of the music videos together and it puts all of these sorts of videos together just because that's sort of the way the people using YouTube behave so that already cleans up a lot of the problem it takes care of the lowest hanging fruit which happens to be a huge one of just managing these millions of videos that's right I remember a few years ago I was talking to someone who was trying to propose that we do a research project concerning people who who are bilingual and this person was making this proposal based on the idea that YouTube could not possibly be good at recommending videos well to people who are bilingual and so she was telling me about this and I said well can you give me an example of what problem do you think we have on YouTube with the recommendations and so she said well I'm a researcher in in the US and and when I'm looking for academic topics I want to look I want to see them in English and so she searched for one found a video and then looked at the watch next suggestions and they were all in in English and so she said oh I see YouTube must think that I speak only English and so she said now I'm actually originally from Turkey and sometimes when I'm cooking let's say I want to make some baklava I really like to watch videos that are in Turkish and so she searched for a video about making the baklava and then and then selected it it was in Turkish and the watch next recommendations were in Turkish and she just couldn't believe how this was possible and how is it that you know that I speak both these two languages and put all the videos together and it's just as a route come of this related graph that's created through collaborative filtering so for me one of my huge interest is just human psychology right and and that's such a powerful platform on which to utilize human psychology to discover what people individual people want to watch next but it's also be just fascinating to me you know I've Google search has ability to look at your own history and I've done that before just just what I've searched three years for many many years and it's fascinating picture of Who I am actually and I don't think anyone's ever summarized that I personally would love that a summary of who I am as a person on the Internet to me because I think it reveals I I think it puts a mirror to me or to others you know that's actually quite revealing and interesting you know just maybe in the number of it's a joke but not really is the of cat videos I've watched videos of people falling you know stuff that's absurd that kind of stuff it's really interesting and of course it's really good for the machine learning aspect to to show to figure out what to show next but it's interesting hey have you just as a tangent played around with the idea of giving a map to people sort of as opposed to just using this information to show us next showing them here are the clusters you've loved over the years kind of thing well we do provide the history of all the videos that you've watched yes so you can definitely search through that and look through it and search through it to see what it is that you've been watching on YouTube we have actually in various times experimented with this sort of cluster idea finding ways to demonstrate or show people what topics they've been interested in or what what clusters they've watched from it's interesting that you bring this up because in some sense the way the recommendation system of YouTube sees a user is exactly as the history of all the videos they've watched on YouTube and so you can think of yourself or any user on YouTube as kind of like a DNA strand of all your videos right that sort of represents you you can also think of it as maybe a vector in the space of all the videos on YouTube and so you know now once you think of it as a vector in the space of all the videos on YouTube then you can start to say okay well you know which videos which which other vectors are close to me and to my vector and and that's one of the ways that we generate some diverse recommendations is because you're like okay well you know these these people seem to be closed with respect to the videos they've watched on YouTube but you know here's a topic or a video that one of them has watched and enjoyed but the other one hasn't that could be an opportunity to make a good recommendation I gotta tell you I mean I know for things that are impossible but I would love to cluster than human beings like I would love to know who has similar trajectories as me you probably would want to hang out alright there's a social aspect there like actually finding some of the most fascinating people I find out in YouTube but have like no followers and I start following them and they create incredible content and you know and on that topic I just love to ask there's some videos just blow my mind in terms of quality and depth and just in every regard are amazing videos and they have like 57 views okay how do you get videos of quality to be seen by many eyes so the measure of quality is it just something yeah how do you know that something is good well I mean I think it depends initially on what sort of video we're talking about so in the realm of let's say you mentioned politics and news in that realm you know quality news or quality journalism relies on having a journalism department right like you you have to have actual journalists and fact-checkers and people like that and so in that situation and in others maybe science or in medicine quality has a lot to do with the authoritative nough sand the credibility and the expertise of the people who make the video now if you're thinking about the other end of the spectrum you know what is the highest quality prank video for what is the highest quality minecraft video yeah right that might be the one that people enjoy watching the most and watch to the end or it might be the one that when we ask people the next day after they watched it were they satisfied with it and so we in in especially in the realm of entertainment have been trying to get at better and better measures of quality or satisfaction or enrichment since I came to YouTube and we started with well you know the first approximation is the one that gets more views but but you know we both know that things can get a lot of views and not really be that high quality especially if people are clicking on something and then immediately realizing that it's not that great and abandoning it and that's why we move from views to thinking about the amount of time people spend watching it what the premise that like you know in some sense the time that someone spends watching a video is related to the value that they get from that video it may not be perfectly related but it has something to say about how much value they get but even that's not good enough right because I myself have spent time clicking through channels on television late at night and ended up watching under siege - for some reason I don't know and if you were to ask me the next day are you glad that you watched that show on TV last night I'd say yeah I wish I would have gone to bed or read a book or almost anything else really and so that's why some people got the idea a few years ago to try to serve at users afterwards and so so we get feedback data from those surveys and then use that in the machine learning system to try to not just predict what you're gonna click on right now what you might watch for a while but what when we ask you tomorrow you'll give four or five stars - so just to summarize what are the signals from a machine learning perspective the user can provide he mentions just clicking on the video views the time watch maybe the relative time watched the clicking like and dislike on the video maybe commenting on the video and those things all of those things and then though the one I wasn't actually quite aware of even though I might have engaged in it is a survey afterwards which is a brilliant idea is there other signals all right I mean that's already a really rich space of signals to learn from is something else well you mentioned commenting also sharing the video if you if you think it's worthy to be shared with someone else you know within YouTube or outside of YouTube as well either let's see you mentioned like dislike yeah like and dislike how important is that it's very important right we want its predictive of satisfaction but it's not it's not perfectly predictive subscribe if you subscribe to the channel of the person who made the video then that also is a piece of information and signals satisfaction although over the years we've learned that people have a wide range of attitudes about what it means to subscribe we would ask some users who didn't subscribe very much why but they watched a lot from a few channels we'd say well why didn't you subscribe and they would say well I I can't afford to pay for anything and you know we tried to let them understand like actually it doesn't cost anything it's free it just helps us know that you are very interested in this creator but then we've asked other people who subscribed to many things and and don't really watch any of the videos from those channels and we say well well why did you subscribe to this if you weren't really interested in any more videos from that channel and they might tell us why just you know I thought the person did a great job and I just want to kind of give him a high five yeah yeah and so yeah that's where I I said I should subscribe to channels where I just this person is amazing I like this person but then I like this person I really want to support them that that's how I click Subscribe right even though I may never actually want to click on their videos when they're releasing it I just love what they're doing and it's maybe outside of my interest area and so on which is probably the wrong way to use the subscribe button but I just want to say congrats this is a great work well so you have to deal with all the space of people that see the subscribe button it's totally different that's right and so you know we we can't just close our eyes and say sorry you're using it wrong you know and we're not gonna pay attention to what you've done we need to embrace all the ways in which all the different people in the world use the subscribe button or the like in the dislike button so in terms of signals of machine learning using for the search and for the recommendation you've mentioned title so like metadata like text data that people provide description and title and maybe keywords so maybe you can speak to the value of those things in search and also this incredible fascinating area of the content itself so the video content itself trying to understand what's happening in the video so YouTube would release a dataset that you know the in the machine learning and computer vision world this is just an exciting space how much is that currently how much he playing with that currently how much is your hope for the future of being able to analyze the content of the video itself well we have been working on that also since I came to YouTube analyzing the content analyzing the content while video right and what I can tell you is that our ability to do it well is still somewhat crude we can we can tell if it's a music video we can tell if it's a sports video we can probably tell you that people are playing soccer we probably can't tell whether it's Manchester United or my daughter's soccer team so these things are kind of difficult and and using them we can use them in some ways so for instance we use that kind of information to understand and inform these clusters that I talked about and also maybe to add some words like soccer for instance to the video if if it doesn't occur in the title or the description which is remarkable that often it doesn't I one of the things that I ask creators to do is is please help us out with the title in the description for instance we were a a few years ago having a live stream of some competition for World of Warcraft on YouTube and it was a very important competition but if you typed World of Warcraft in search you wouldn't find it well the Warcraft wasn't in the title World of Warcraft wasn't in the title it was match four seven eight you know a team versus B team and World of Warcraft wasn't the title yes like come on give me being literal being literal on the Internet is actually very uncool which is the problem oh is that right well I mean in some sense well some of the greatest videos I mean there's a humor to just being indirect being witty and so on and actually being you know machine learning algorithms want you to be you know literal right usually want to say what's in the thing be very very simple and in in some sense that gets away from wit and humor so you have to play with both right so but you're saying that for now sort of the content of the title the content of the description the actual text is is one of the best ways to uh for the for the algorithm to find your video and put them in the right cluster that's right and and I would go further and say that if you want people human beings to select your video in search then it helps to have let's say World of Warcraft in the title because why would a person's you know if they're looking at a bunch they type World of Warcraft and they have a bunch of videos all of whom say World of Warcraft except the one that you uploaded well even the person is gonna think well maybe this isn't some house search made a mistake this isn't really about World of Warcraft so it's important not just for the machine learning systems but also for the people who might be looking for this sort of thing they get a clue that it's what they're looking for by seeing that same thing prominently in the title of the video okay let me push back on that so I think from the algorithmic perspective yes but if they typed in World of Warcraft and saw a video that with the title simply winning and and and the thumbnail has like a sad orc or something I don't know right like I think that's much it's Iraq it gets your curiosity up and then if they could trust that the algorithm was smart enough to figure out somehow that this is indeed a World of Warcraft video that would have created the most beautiful experience I think in terms of just the wit and the humor and the curiosity that we human beings actually have but you're saying I mean realistically speaking is really hard for the algorithm to figure out that the content of that video will be a world of warcraft and you have to accept that some people are gonna skip it yeah right I mean and so you're right the people who don't skip it and select it are gonna be delighted yeah but other people say might say but yeah this is not what I was looking for and making stuff discoverable I think is what you're really working on and hoping so yeah so from your perspective to put stuff in the description and remember the collaborative filtering part of the system it starts by the same user watching videos together right so the way that they're probably going to do that is by searching for them that's a fascinating aspect it's like ant colonies that's how they find stuff is so I mean you would agree for collaborative filtering in general is one curious ant one curious user essential sort of just a person who is more willing to click on random videos and sort of explore these cluster spaces in your sense how many people are just like watching the same thing over and over and over and over and how many are just like the explorers I just kind of like click on stuff and then help help the other ant and the ants colony discover the cool stuff do you have a sense of that or no I really don't think I have a sense me OK relative sizes of those groups but I but I would say that you know people come to YouTube with some certain amount of intent and as long as they to the extent to which they they try to satisfy that intent that certainly helps our systems right because our systems rely on on kind of a faithful amount of behavior the right like and there are people who try to trick us right there are people and machines that try to associate videos together that really don't belong together but they're trying to get that Association made because it's profitable for them and so we have to always be resilient to that sort of attempt at gaming the system so speaking to that there's a lot of people that in a positive way perhaps I don't know I I don't like it but I like to gain want to try to gain the system to get more attention and everybody creators in a positive sense want to get attention right so how do you how do you work in this space when people create more and more sort of click Beatty titles and thumbnails sort of a very tasking derek has made a video it basically describes that it seems what works is to create a high quality video really good video what people would want to watch and wants to click on it but have clicked BT titles and thumbnails to get him to click on it in the first place and he's saying I'm embracing this bactrim just gonna keep doing it and I hope you forgive me for doing it and you will enjoy my videos once you click on them so in what sense do you see this kind of clickbait style attempt to manipulate to get people in the door to manipulate the algorithm or play with the algorithmic game the algorithm I think that that you can look at it as an attempt to game the algorithm but even if you were to take the algorithm out of it and just say ok well all these videos happen to be lined up which the algorithm didn't make any decision about which one to put at the top or the bottom but they're all lined up there which one are the people going to choose and and I'll tell you the same thing that I told Derek is you know I have a bookshelf and they have two kinds of books on them science books I have my math books from when I was a student and they all look identical except for the titles on the covers they're all yellow they're all from Springer and they're every single one of them the cover is totally the same yes right yeah on the other hand I have other more pop science type books and they all have very interesting covers right and they have provocative titles and things like that I mean I wouldn't say that they're clickbait II because they are indeed good books and I don't think that they cross any line but but you know the that's just a decision you have to make right like the people who who write classical recursion theory by pure OD Freddie he was fine with the yellow title and the and nothing more whereas I think other people who who wrote a more popular type book understand that they need to have a compelling cover and a compelling title and and you know I don't think there's anything really wrong with that we do we do take steps to make sure that there is a line that you don't cross and if you go too far maybe your thumbnails especially racy or or you know it's all cats with too many exclamation points we observe that users are kind of you know sometimes offended by that and so so for the users who were offended by that we will then depress or suppress those videos and which reminds me that there's also another signal where users can say I don't know if was recently added but I really enjoy it just saying I don't I didn't something like I I don't want to see this video anymore or something like like this is a like there's certain videos just cut me the wrong way like just just jump out at music I don't wanna I don't want this and it feels really good to clean that out to be like I don't that's not that's not for me I don't know I think that might have been recently added by this that's also a really strong signal yes absolutely right we don't want to make a recommendation that people are unhappy with and that makes me that particular one makes me feel good as a user in general and as a machine learning person because I feel like I'm helping the algorithm my interaction I need you don't always feel like I'm helping the algorithm like I'm not reminded of that fact like for example Tesla and Otto Pollan you know on musk create a feeling for their customers for people their own test is that there helping the algorithm of testify like they're all like a really proud they're helping nicely learn I think YouTube doesn't always remind people that you're helping the algorithm get smarter and for me I love that idea like we're all collaboratively like Wikipedia gives that sense they were all together creating a beautiful thing YouTube is uh doesn't always remind me of that it's uh this conversation is Right any of that but well that's a good tip we should keep that fact in mind when we design these features well I I'm not sure I I really thought about it that way but that's a very interesting perspective it's an interesting question of personalization that I feel like when I click like on a video I'm just improving my experience it would be great it would make me personally people are different but make me feel great if I was helping also the YouTube algorithm broadly say something you know saying like there's a that I don't know if that's human nature we you want the products you love and I certainly love YouTube like you want to help it get smarter and smarter smarter because there's some kind of coupling between our lives together being better if if YouTube was better than I will my life will be better and that's that kind of reasoning I'm not sure what that is and I'm not sure how many people share that feeling it could be just a machine learning feeling but at that point how much personalization is there in terms of next video recommendations so is it kind of all really boiling down to a clustering like you find in ears clusters to me and so on and that kind of thing or how much is person s to me the individual completely it's very very personalized so your experience will be quite a bit different from anybody else's who's watching that same video at least when they're logged in and the reason is is that we found that that users often want two different kinds of things when they're watching a video sometimes they want to keep watching more on that topic or more in that genre and other times they just are done and they're ready to move on to something else and so the question is well what is this something else and one of the first things one can imagine is well maybe something else is the latest video from some channel to which you've subscribed and that's gonna be very different from for you than it is for me right and and even if it's not something that you subscribe to it's something that you watch a lot and again that'll be very different on a person-by-person basis and so even the watch next as well as the homepage of course is quite personalized so what we met some of the signals but what a success look like what a success look like in terms of the algorithm creating a great long-term experience for a user or put another way if you look at the videos I've watched this month how do you know the algorithm succeeded for me I think first of all if you come back and watch more YouTube then that's one indication that you've found some value from it so just the number of hours is a powerful indicator well I mean not the hours themselves but the fact that you return on another day so that's probably the most simple indicator people don't come back to things that they don't find value in right there's a lot of other things that they could do but like I said I mean ideally we would like everybody to feel that YouTube enriches their lives and that every video they watched is the best one they've ever watched since they've started watching YouTube and so that's why we survey them and ask them like is this one to five stars and so our version of success is every time someone takes that survey they say it's five stars and if we ask them is this the best video you've ever seen on YouTube they say yes every single time so it's hard to imagine that we would actually achieve that maybe asymptotically we would get there but but that would be what we think success is it's funny have recently said some way I don't know maybe tweeted but that Ray Dalio has this video on the economic machine I forget what it's called but it's a 30-minute video and I said it's the the greatest video I've ever watched on YouTube it's it's like I watched the whole thing and my mind was blown is a very crisp clean description of how the at least the American economic system works it's a beautiful video and I was just I wanted to click on something to say this is the best thing this is the best thing ever please let me I can't believe I discovered it I mean the the views and the likes reflect its quality but I was almost upset that I haven't found it earlier and wanted to find other things like it I don't think I've ever felt that this is the best video ever and that was that and to me the ultimate utopia the best experiences were every single video where I don't see any of the videos I regret in every single video I watch is one that actually helps me grow helps me enjoy life be happy and so on well so that's that's that's a heck of uh the thought that's one of the most beautiful and ambitious I think machine learning tasks so when you look at a society as opposed to any individual user do you think of how YouTube is changing society when you have these millions of people watching videos growing learning changing having debates do you have a sense of yeah what the big impact on society is because I think it's huge but you have a sense of what direction we're taking this world well I mean I think you know openness has had an impact on society already there's a lot of what do you mean by openness well the fact that unlike other mediums there's not someone sitting at YouTube who decides before you can upload your video whether it's worth having you uploaded or worth anybody seeing it really right and so you know there are some creators who say like I I wouldn't have this opportunity to to reach an audience Tyler Oakley often said that you know he wouldn't have had this opportunity to reach this audience if it weren't for YouTube and and so I think that's one way in which YouTube has changed Society I know that there are people that I work with from outside the United States especially from places where literacy is low and they think that YouTube can help in those places because you don't need to be able to read and write in order to learn something important for your life maybe you know how to do some job or how to fix something and so that's another way in which I think YouTube is possibly changing society so I've worked at YouTube for eight almost nine years now and it's fun because I meet people and you know you tell them where they where you work you say you work on YouTube and they immediately say I love you too Yeah right which is great makes me feel great but then of course when I ask them well what is it that you love about YouTube not one time ever has anybody said that the search works outstanding or that the recommendations are great what they always say when I ask them what do you love about YouTube is they immediately start talking about some channel or some creator or some topic or some community that they found on YouTube and that they just loved yeah and so that has made me realize that YouTube is really about the video and connecting the people with the videos and then everything else kind of gets out of the way so beyond the video it's an interesting because you kind of mentioned creator what about the connection with just the individual creators as opposed to just individual video so like I gave the example of Ray Dalio video that the video itself is incredible but there's some people or just creators that I love that they're one of the cool things about people who call themselves youtubers or whatever is they have a journey they usually almost all of them are hurt they suck horribly in the beginning and then they kind of grow you know and then there's that genuineness in their growth so you know YouTube clearly wants to help creators connect with their audience in this kind of way so how do you think about that process of helping creators grow helping the connect with their audience develop not just individual videos but the entirety of a creators life on YouTube well I mean we're trying to help creators find the biggest audience that they can find and the reason why that's you you brought up creator versus video the reason why creator channel is so important is because if we have a hope of of people coming back to YouTube well they have to have in their minds some sense of what they're gonna find when they come back to YouTube if YouTube were just the next viral video and I have no concept of what the next viral video could be one time it's a cat playing a piano and the next day it's some children interrupting a reporter and the next day it's you know some other thing happening then then it's hard for me to to when I'm not watching YouTube say gosh I really you know would like to see something from someone or about something right and so that's why I think this connection between fans and creators so important for both because it's it's a way of a sort of fostering a relationship that can play out into the future let me talk about kind of a dark and interesting question in general and again a topic that you or nobody has an answer to but social media has a sense of you know it gives us highs and gives us lows in the sense that so creators often speak about having sort of burn burn out and having psychological ups and and challenges mentally in terms of continuing the creation process there's a momentum there's a huge excited audience that makes everybody feel that makes creators feel great and I think it's more than just financial I think it's literally just they love that sense of community it's part of the reason I upload to YouTube I don't care about money never well what I care about is the community but some people feel like this momentum and even when there's times in their life when they don't feel you know the for some reason don't feel like creating so how do you think about burnout this mental exhaustion that some YouTube creators go through that's something we have an answer for is that something how do we even think about that well the first thing is we want to make sure that the YouTube systems are not contributing to this sense right and so we've done a fair amount of research to demonstrate that you can absolutely take a break if you are a creator and you've been uploading a lot we have just as many examples of people who took a break and came back more popular than they were before as we have examples of going the other way yeah can we pause on that for a second so the feeling that people have I think is if I take a break everybody well the party will leave right so if you can just linger on that so in your sense that taking a break is okay yes taking a break is absolutely okay and the reason I say that is because we have we can observe many examples of being of creators coming back very strong and even stronger after they have taken some sort of break and so I just want to dispel the myth that this somehow necessarily means that your channel is gonna go down or lose views that is not the case we know for sure that this is not a necessary outcome and so we we want to encourage people to make sure that they take care of themselves that is job one right you you have to look after yourself and your mental health and you know I think that it probably in some of these cases contributes to better videos once they come back right because a lot of people I mean I know myself if I'm burn out on something that I'm probably not doing my best work even though I can keep working until I pass out and so I think that the the taking a break may even improve the creative ideas that someone has okay I think it's a really important thing to sort of to dispel I think it applies to all of social media like literally I've taken a break for a day every once in a while sorry sorry that sounds like a short time but even like sorry email just taking a break from email or only checking email once a day especially when you're going through something psychologically in your personal life or so on or really not sleeping much because it work deadlines it can refresh you in a way that's that's profound and so the same applies there when you came back right it's there and it looks different actually when you come back you sort of brighter I'd some coffee everything the world looks better so it's important to take a break when you need it so you've mentioned kind of the the YouTube algorithm isn't you know e equals MC squared is that's a single equation it's it's potentially sort of more than a million lines of code sort of is it more akin to what autonomous successful autonomous vehicles today are which is they're just basically patches on top of patches of heuristics and human experts really tuning the algorithm and have some machine learning modules or is it becoming more and more a giant machine learning system with humans just doing a little bit of tweaking here and there what's your sense first of all do you even have a sense of what is the YouTube algorithm at this point and whichever however much you do have a sense what does it look like well we don't usually think about it as the algorithm because it's a bunch of systems that work on different services the other that I think people don't understand is that what you might refer to as the YouTube algorithm from outside of YouTube is actually a you know a bunch of code and machine learning systems and heuristics but that's married with the behavior of all the people who come to YouTube every day so the people part of the code Accession exactly right like if there were no people who came to youtube tomorrow then there the algorithm wouldn't work anymore right so that's a critical part of the algorithm and so when people talk about well the algorithm does this the algorithm does that it's sometimes hard to understand well you know it could be the the viewers are doing that and the algorithm is mostly just keeping track of what the viewers do and then reacting to those things in in sort of more fine-grained situations and I and I think that this is the way that the recommendation system and the search system and and probably many machine learning systems evolve is you know you start trying to solve a problem and the first way to solve a problem is often with a simple heuristic right and and you know you want to say what are the videos we're gonna recommend well how about the most popular ones weighted that's where you start and and over time you collect some data and you refine your situations so that you're making less heuristics and you're you're building a system that can actually learn what to do in different situations based on some observations of those situations in the past and and you keep chipping away at these heuristics over time and so I think that just like with diversity you know I think the first diversity measure we took was okay not more than three videos in a row from the same Channel right it's a pretty simple heuristic to encourage diversity it worked right you needs to see four or five six videos in a row from the same Channel and over time we try to chip away at that it and make it more fine-grain and basically have it remove the heuristics in favor of something that can react to individuals and individual situations so how do you you mentioned you know we we know that something worked how do you get a sense when decisions of a kind of a be testing that this idea was a good one this was not so good what's how do you measure that and across which time scale across how many users that kind of that kind of thing well you mentioned that a B experiments and so just about every single change we make to YouTube we do it only after we've run a a B experiment and so in those experiments which run from one week to months we measure hundreds literally hundreds of different variables and and measure changes with confidence intervals in all of them because we really are trying to get a sense for ultimately does this improve the experience for viewers that's the question we're trying to answer and an experiment is one way because we can see certain things go up and down so for instance if we noticed in the experiment people are dismissing videos less frequently or they're saying that they're more satisfied they're giving more videos five stars after they watch them then those would be indications of that the experiment is successful that it's improving the situation for viewers but we can also look at other things like we might do user studies where we invite some people in and ask them like what do you think about this what do you think about that how do you feel about this and other various kinds of user research but ultimately before we launch something we're gonna want to run an experiment so we get a sense for what the impact is going to be not just to the viewers but also to the different channels and all of them an absurd question nobody know what actually is interesting maybe there's an answer but if I want to make a viral video how do I do it I don't know how you make a viral video I I know that we have in the past tried to figure out if we could detect when a video video was going to go viral you know and those were you take the first and second derivatives of the view count and maybe use that to do some prediction but but I can't say we ever got very good at that oftentimes we look at where the traffic was coming from you know if it's if it's a lot of the viewership is coming from something like Twitter then then maybe it has a higher chance of becoming viral than maybe if then if it were coming from search or something but that was just trying to detect a video that might be viral how to make one like I have no idea so yeah you get your kids to interrupt you while you're on the news on the news absolutely as but after the fact on a one individual video so the head of time predicting is a really hard task but after the video went viral in analysis can you sometimes understand why I went viral from the perspective of YouTube broadly first I was even interesting for YouTube that a particular videos viral or is does that not matter for the individual for the experience of people well I think people expect that if a video video is going viral and it's something they would be interested in then I wouldn't I think they would expect YouTube to recommend it to them right um so someone's going viral it's good to just let the wave ride the wave of its violence well I mean we want to meet people's expectations in that way of course so like like I mentioned I hung out with Derek Muller a while ago a couple of months back he's actually the person who suggested I talk to you on this podcast all right well thank you Derek at that time he just recently posted an awesome science video titled why are ninety-six million black balls on this reservoir and in a matter of I don't know how long but like a few days he got thirty million views and it's still growing is this something you can analyze and understand why it happened this video and you won't particularly like it I mean we can surely see where it was recommended where it was found who watched it and those sorts of things so it's actually sorry to interrupt it is the video which helped me discover who Derek is I didn't know who he is before so I remember you know usually I just have all of these technical boring MIT Stanford talks in my recommendation because that's how I watch and then all sudden there's this black balls in reservoir video with like an excited nerd in the would like just and why is this being recommended to me so I close down and watch the whole thing it was awesome but and a lot of people had that experience like why was I recommend this but they all of course watched it and enjoyed it which is what's your sense of this just wave of recommendation and that comes with this viral video that ultimately people get enjoy after they click on it well I think it's the system you know basically doing what anybody who's recommending something would do which is you show it to some people and if they like it you say okay well can I find some more people who are a little bit like them okay I'm gonna try it with them oh they like it too let me expand the circle some more find some more people oh it turns out they like it too so can you just keep going until you get some feedback that says no now you've gone too far these people don't like it anymore and so I think that's basically what happened now you asked me about how to make a video go viral or make a viral video I don't think that if you or I decided to make a video about 96 million balls that it would also go viral it's possible that Derek made like um the canonical video about those black balls yeah lake and so he did actually right and and so I don't know whether or not just following along is the secret yeah but it's fascinating I mean just like you said the algorithm sort of expanding that circle and then figuring out that more and more people did enjoy and that sort of phase shift of just a huge number of people enjoying in the algorithm quickly automatically I assume figuring that out that's a I don't know the dynamics in psychology that is a beautiful thing and so what do you think about the idea of of clipping like and too many people annoyed me into doing it which is they were requesting it I said very beneficial to add clips in like the the coolest points and actually have explicit videos like I'm reapplying a video like a short clip which is what the the podcasts are doing yeah do you see as opposed to like I also add time stamps for the topics no people want the clip do you see YouTube somehow helping creators with that process or helping connect clips to the original videos what is that just in a long list of amazing things to work towards yeah I mean it's not something that I think we've we've done yet but I can tell you that I think clipping is great and I think it's actually great for you as a creator and here's the reason if you think about I mean let's let's say the NBA is uploading videos of of its games well people might search for warriors vs. rockets or they might search for Steph Curry and so a highlight from the game in which Steph Curry makes an amazing shot is an opportunity for someone to find a portion of that video and so I think that you never know how people are gonna search for something that you've created and so you wanna I would say you want to make clips and and add titles and things like that so that they can find it as easily as possible do you have a dream of a future perhaps a distant future when the YouTube algorithm figures that out sort of automatically detects the parts of the video that are really interesting exciting potentially exciting for people and sort of clip them out in this incredibly rich space if you talk about if you thought even just this conversation we probably covered 30 40 little topics and there's a huge space of users that would find you know 30 percent of those topics interesting and that's is very different it's something that's beyond my ability to clip out right but the algorithm might be able to figure all that out sort of expand into clips do you ever you think about this kind of thing do you have a hope a dream that one day the album will be able to do that kind of deep content analysis well we've actually had projects that attempt to achieve this but it really does depend on understanding the video well and our understanding of the video right now is quite crude and so I think it would be especially hard to do it with a conversation like this one might be able to do it with let's say a soccer match more easily right you could probably find out where the goals were scored and then of course you you need to figure out who it was that scored the goal and and that might require human to do some annotation but I think that trying to identify coherent topics in a transcript like like the one of our conversation is is not something that we're gonna be very good at right away and I was speaking more to the general problem actually of being able to do both a soccer match and our conversation without explicit sort of almost my hope was that there exists an algorithm that's able to find exciting things in video so Google now on Google search will help you find the segment of the video that you're interested in so if you search for something like how to change the filter in my dishwasher then if there's a long video about your dishwasher and this is the part where the person shows you how to change the filter then then it will highlight that area and provide a link directly to it and you know if from your recollection do you know if the thumbnail reflects like what's the difference between showing the full video and the shorter clip do you know what how its presented in search results don't remember how its presented and the other thing I would say is that right now it's based on creator annotations got it so it's not the thing I'm talking about but there but but folks are working on the more automatic version it's interesting people might not imagine this but a lot of our systems start by using almost entirely the audience behavior and then as they get better the refinement comes from using the content and I wish and I know there's privacy concerns but I wish YouTube explored the space which is sort of putting a camera on the user's if they allowed it right to study there like I did a lot of emotion recognition work and so on to study actual sort of rich or signal one of the cool things when you upload 360 like VR video to YouTube and I've done this a few times so I've uploaded myself it's a horrible idea some people enjoyed it but whatever the video of me giving a lecture in 360 over 360 camera it's cool because YouTube allows you to then watch where did people look at there's a heat map of where you know avoid the center of the VR experience was and it's interesting because that reveals to you like what people looked at and it's it's very not always what you were though it's not in the case of the lecture is pretty boring it is what we're expecting but we did a few funny videos where there's a bunch of people doing things and they everybody tracks those people you know in the beginning they all look at the main person and they start spreading around and looking into other people it's fascinating so that kind of that's a really strong signal of what people found exciting in the video I don't know how you get that from people just watching except they tuned out at this point like it's hard to measure this moment was super exciting for people I don't know how you get that signal maybe comment is there a way to get that signal where this was like this is when their eyes opened up they're like like for me with the Ray Dalio video right like first I was like okay this is another one of these like dumb it down for you videos and then you like start watching it's like okay there's really crisp clean deep explanation of how the economy works that's where I like set up and started watch right at that moment is there a way to detect that the only way I can think of is by asking people to just label it yeah you mentioned that we're quite far away in terms of doing video analysis deep video analysis ago of course Google YouTube you know we're quite far away from solving autonomous driving problem - yes I don't know I think we're closer to that what the you know you never know and the Wright brothers thought they're never they're not gonna five fifty years three years before they flew so what are the biggest challenges would you say is it the broad challenge of understanding video understanding natural language understanding the the challenge before the entire machine learning community or just being able to understand data is there something specific to video that's even more challenging than an understanding natural language understanding what's your sense of what the biggest video is just so much information and so precision becomes a real problem it's like a you know you're trying to classify something and you've got a million classes and you the distinctions among them at least from a from a machine learning perspective are often pretty small right like you know you need to see this person's number in order to know which player it is and and there's a lot of players or you need to see you know the logo on their chest in order to know like which which team they play for and so and that's just figuring out who's who right and then you go further and saying okay well you know was that a goal was it not a goal like is that an interesting moment as you said or is that not an interesting moment these things can be pretty hard so okay so yawn laocoön I'm not sure if you're familiar sort of with his current thinking and work so he believes that self what is referring to self supervised learning will be the solution sort of to achieving this kind of greater level of intelligence in fact the thing he's focusing on is watching video and predicting the next frame so predicting the future of video right so for now we're very far from that but his thought is because it's unsupervised uh-huh or is it here first to a self supervise you know if you watch enough video essentially if you watch YouTube you'll be able to learn about the nature of reality the physics the common sense reasoning required by just teaching a system to predict the next frame so he's confident this is the way to go so see you from the perspective of just working with this video how do you think an algorithm that just watches all of YouTube stays up all day and night watching YouTube will be able to understand enough of the physics of the world about the way this world works failed to do common-sense reasoning and so on well I mean we have systems that already watch all the videos on YouTube right but they're just looking for very specific things right they're supervised learning systems that are trying to identify something or classify something and I don't know if I don't know if predicting the next frame is really gonna get there because I don't I'm not an expert on compression algorithms but I understand that that's kind of what compression video compression algorithms do is they basically try to predict the next frame and and and then fix up the places where they got it wrong and that leads to higher compression and if you actually put all the bits for the next frame there so so I I don't know if I believe that just being able to predict the next frame is gonna be enough because because there's so many frames and even a tiny bit of error on a per frame basis can lead wildly different videos so the thing is the idea of compression is one way to do compression is to describe through text with containing the video that's the ultimate high level of compression so the idea is tradition when you think of video image compression you're trying to maintain the same visual quality while reducing the size but if you think of deep learning from a bigger perspective what compression is is you're trying to summarize the video and the idea there is if you have a big enough neural network this by watching the next bit trying to predict the next frame you'll be able to form a compression of actually understanding what's going on in the scene if there's two people talking you can just reduce that entire video and into the fact that two people are talking and maybe the content of what they're saying and so on that that's kind of the the open-ended dream so I just wanted to sort of express it because it's interesting compelling notion but it is nevertheless true that video our world is a lot more complicated than we getting credit for I mean in terms of search and discovery we have been working on trying to summarize videos in text or or with some kind of labels for eight years at least and we're kind of so so so and so if you would say it's the problem is a hundred percent solved and eight years ago was zero percent solved how where are we on that timeline would you say yeah to summarize a video well maybe less than a quarter of the way so on that topic what does YouTube look like ten twenty thirty years from now I mean I think that YouTube is evolving to take the place of TV you know I grew up as a kid in the 70s and I watched a tremendous amount of television and I feel sorry for my poor mom because people told her at the time that it was going to rot my brain and that she should kill her television but anyway I mean I think that YouTube is at least for my family a better version of television right it's one that is on demand it's more tailored to the things that my kids want to watch and also they can find things that they would never have found on television and so I think that at least from just observing my own family that's where we're headed is that people watch YouTube kind of in the same way that I watch television when I was younger so from a search and discovery perspective what do you what are you excited about and then the 5 10 20 30 years like what kind of things it's already really good I think it's achieved a lot of of course we don't know what's possible so it's a it's the the task of search of typing in the text or discovering new videos by the next recommendation I personally I'm really happy with the experience that continuously I rarely watch a video that's not awesome from my own perspective but what's what's else is possible what are you excited about well I think introducing people to more of what's available on YouTube is not only very important to YouTube in to creators but I think it will help enrich people's lives because there's a lot that I'm still finding out is available on YouTube that I didn't even know I've been working YouTube eight years and it wasn't until last year that I learned that that I could watch USC football games from the 1970s no like I didn't even know that was possible last year and I've been working there quite some time so you know what was broken about about that but it took me seven years to learn that this stuff was already on YouTube even when I got here so I think there's a big opportunity there and then as I said before you know we want to make sure that YouTube finds a way to ensure that it's acting responsibly with respect to society and enriching people's lives so we want to take all of the great things that it does and make sure that we are eliminating the negative consequences that might happen and then lastly if we could get to a point where all the videos people watch are the best ones they've ever watched that would be outstanding to do you see in many senses becoming a window into the world for people and it's especially with live video you get to watch events I mean it's really it's the way you experience a lot of the world that's out there is better than TV in many many ways so do you see becoming more than just video do you see creators creating visual experiences and virtual worlds so if I'm talking crazy now but sort of virtual reality and entering that space there's that at least for now totally outside of what YouTube is thinking about I mean I think Google is thinking about virtual reality I don't think about virtual reality too much um I know that we would want to make sure that YouTube is there when virtual reality becomes something or if virtual reality becomes something that a lot of people are interested in but I haven't seen it really take off yet take off well the the future is wide open christos I've been really looking forward to this conversation has been a huge honor thank you for answering some of the more difficult questions I've asked I'm really excited about what YouTube has in store for us it's one of the greatest products of ever use and continues so thank you so much for talking it it's my pleasure thanks for asking me thanks for listening to this conversation and thank you to our presenting sponsor cash app downloaded use code Lex podcast you'll get ten dollars and ten dollars will go to first a stem education nonprofit that inspires hundreds of thousands of young minds to become future leaders and innovators if you enjoy this podcast subscribe on YouTube give it five stars an apple podcast follow on Spotify supported on patreon or simply connect with me on Twitter now let me leave you with some words of wisdom from Marcel Proust the real voyage of discovery consists not in seeking new landscapes but in having new eyes thank you for listening I hope to see you next time you