Jitendra Malik: Computer Vision | Lex Fridman Podcast #110
the following is a conversation with jitendra malik a professor at berkeley and one of the seminal figures in the field of computer vision the kind before the deep learning revolution and the kind after he has been cited over 180 thousand times and has mentored many world-class researchers in computer science quick summary of the ads two sponsors one new one which is better help and an old goody expressvpn please consider supporting this podcast by going to betterhelp.com lex and signing up at expressvpn.com lexpod click the links buy the stuff it really is the best way to support this podcast and the journey i'm on if you enjoy this thing subscribe on youtube review it with 5 stars on apple podcast support it on patreon or connect with me on twitter at lex friedman however the heck you spell that as usual i'll do a few minutes of ads now and never neons in the middle that can break the flow of the conversation this show is sponsored by better help spelled h-e-l-p help check it out at betterhelp.com lex they figure out what you need and match you with a licensed professional therapist in under 48 hours it's not a crisis line it's not self-help it's professional counseling done securely online i'm a bit from the david goggins line of creatures as you may know and so have some demons to contend with usually on long runs or all nights working forever and possibly full of self-doubt it may be because i'm russian but i think suffering is essential for creation but i also think you can suffer beautifully in a way that doesn't destroy you for most people i think a good therapist can help in this so it's at least worth a try check out their reviews they're good it's easy private affordable available worldwide you can communicate by text anytime and schedule weekly audio and video sessions i highly recommend that you check them out at betterhelp.com lex this show is also sponsored by expressvpn get it at expressvpn.com to support this podcast and to get an extra three months free on a one-year package i've been using expressvpn for many years i love it i think expressvpn is the best vpn out there they told me to say it but it happens to be true it doesn't log your data it's crazy fast and it's easy to use literally just one big sexy power on button again for obvious reasons it's really important that they don't log your data it works on linux and everywhere else too but really why use anything else shout out to my favorite flavor of linux ubuntu mate 2004 once again get it at expressvpn.comlexpod to support this podcast and to get an extra three months free and a one year package and now here's my conversation with jitendra in 1966 seymour papper at mit wrote up a proposal called the summer vision project to be given as far as we know to 10 students to work on and solve that summer so that proposal outlined many of the computer vision tasks we still work on today why do you think we underestimate and perhaps we did underestimate and perhaps still underestimate how hard computer vision is because most of what we do in vision we do unconsciously or subconsciously in human vision in human vision so that gives us this that effortlessness gives us the sense that oh this must be very easy to implement on a computer now this is why the early researchers in ai got it so wrong however if you go into neuroscience or psychology of human vision then the complexity becomes very clear the fact is that a very large part of the the cerebral cortex is devoted to visual processing i mean and this is true in other primates as well so once we looked at it from a neuroscience or psychology perspective it it becomes quite clear that the problem is very challenging and it will take some time you said the higher level parts are the harder parts i think vision appears to to be easy because uh most of what visual processing is subconscious or unconscious right so we underestimate the difficulty whereas uh when you are like proving a mathematical theorem or playing chess the difficulty is much more evident so because it is your conscious brain which is processing uh various aspects of the problem-solving behavior whereas in vision all this is happening but it's not in your awareness it's in your it's operating below that but it's it still seems strange yes that's true but it seems strange that as computer vision researchers for example the community broadly is time and time again makes the mistake of um thinking the problem is easier than it is or maybe it's not a mistake we'll talk a little bit about autonomous driving for example how hard of a vision task that is it do do you think i mean what is it just human nature or is there something fundamental to the vision problem that we we underestimate we're still not able to be cognizant of how hard the problem is yeah i think in the early days it could have been excused because in the early days all aspects of ai were regarded as too easy but i think today it is much less excusable and i think why people fall for this is because of what i call the fallacy of the successful first step there are many problems in vision where getting 50 of the solution you can get in one minute getting to 90 percent can take you a day getting to 99 percent may take you five years and 99.99 may be not in your lifetime i wonder if that's a unique division that it seems that language people are not so confident about so natural language processing people are a little bit more cautious about our ability to to solve that problem i think for language people intuit that we have to be able to do natural language understanding for vision it seems that we're not cognizant or we don't think about how much understanding is required it's probably still an open problem but in your sense how much understanding is required to solve vision like this put another way how much something called common sense reasoning is required to really be able to interpret even static scenes yeah so vision operates at uh at all levels and there are parts which are which can be solved with what we could call maybe peripheral processing so in the in the human vision literature there used to be these terms sensation perception and cognition which roughly speaking referred to like the front end of processing middle stages of processing and higher level of processing and i think they made a big deal out of out of this and they wanted to just study only perception and then dismiss certain certain problems as being quote cognitive but really i think these are artificial divides the problem is continuous at all level and there are challenges at all levels the techniques that we have today they work better at the lower and mid levels of the problem i think the higher levels of the problem quote the cognitive levels of the problem are there and we in many real applications we have to confront them now how much that is necessary will depend on the application for some problems it doesn't matter for some problems it matters a lot so i am for example a pessimist on fully autonomous driving in the near future and the reason is because i think there will be that 0.01 percent of the cases where quite sophisticated cognitive reasoning is called for however there are tasks where you can first of all they are much more they are robust so in the sense that error rates error is not so much of a problem for example uh uh let's say we are you're doing uh image search you're trying to get images based on some some some description some visual description we are very tolerant of errors there right i mean when google image search gives you some images back and a few of them are wrong it's okay it doesn't hurt anybody there's no there's not a matter of life and death but making mistakes when you're driving at 60 miles per hour and you could potentially kill somebody is much more important so just for the for the fun of it since you mentioned let's go there briefly about autonomous vehicles so one of the companies in the space tesla is work with andre karpathy and elon musk are working on a system called autopilot which is primarily a vision-based system with eight cameras and uh basically a single neural network a multi-task neural network they they call it hydro net multiple heads so it does multiple tasks but is forming the same representation at the core do you think driving can be converted in this way to uh purely a vision problem and then solved within you with learning or even more specifically in the current approach what do you think about what tesla autopilot team is doing so the way i think about it is that there are certainly subset subsets of the visual based driving problem which are quite solvable so for example driving in freeway conditions is quite a solvable problem i think there were demonstrations of that going back to the 1980s by someone called ernst stickmans in munich in the 90s there were approaches from carnegie mellon there were approaches from our team at berkeley in the 2000s there were approaches from stanford and so on so autonomous driving in certain settings is very doable the challenge is to have an autopilot work under all kinds of driving conditions at that point it's not just a question of vision or perception but really also of control and dealing with all the edge cases so where do you think most of the difficult cases to me even the highway driving is an open problem because uh it applies the same 50 90 95 99 rule or the first step the fallacy of the first step i forget how you put it we fall victim to i think even highway driving has a lot of elements because to solve autonomous driving you have to completely relinquish the the fat help of a human being you're always in control so that you're really going to feel the edge cases so i i think even highway driving is really difficult but in terms of the general driving task do you think vision is the fundamental problem or is it also your action the the interaction with the environment the ability to uh and then like the middle ground i don't know if you put that under vision which is trying to predict the behavior of others which is a little bit in the world of understanding the scene but it's also trying to form a model of the actors in the scene and predict their behavior yeah i include that in vision because to me perception blends into cognition and building predictive models of other agents in the world which could be other agents could be people other agents could be other cars that is part of the task of perception because perception always has to uh not tell us what is now but what will happen because what's now is boring it's done it's over with okay yeah we care about the future because we act in the future and we care about the past and as much as it informs what's going to happen in the future so i think we have to build predictive models of of of behaviors of people and and those can get quite complicated so uh uh i mean uh i i've seen examples of this in uh actually i mean i own a tesla and it has various safety features built in and uh what i see are these examples where let's say there is some uh skateboarder i mean this i and i i don't want to be too critical because obviously this is these are the systems are always being improved and any specific criticism i have maybe the system six months from now will not have that that that particular failure mode so uh it it had it it had the wrong response and it's because it couldn't predict what what this skateboarder was going to do okay and because it really required that higher level cognitive understanding of what skateboarders typically do as opposed to a normal pedestrian so what might have been the correct behavior for a pedestrian a typical behavior for pedestrian was not the typical behavior for a skateboarder right yeah and uh so so therefore to do a good job there you need to have enough data where you have pedestrians you also have skateboarders you've seen enough skateboarders to see what uh what kinds of patterns or behavior they have so it is it is in principle with enough data that problem could be solved but uh i think our current systems computer vision systems they need far far more data than humans do for learning those same capabilities so say that there is going to be a system that solves autonomous driving do you think it will look similar to what we have today but have a lot more data perhaps more compute but the fundamental architectures involved like neuro well in the case of tesla autopilot is neural networks do you think it will look similar in that regard and we'll just have more data that's a scientific hypothesis as which way is it going to go uh i will tell you what i would bet on uh so and this is at my general philosophical position on how these uh learning systems have been uh what we have found currently very effective in computer vision uh with in in the deep learning paradigm is sort of tabula rasa learning and tabular us are learning in a supervised way with lots and lots of what's going on in the sense that blank slate we just have the system which is given a series of experiences in this setting and then it learns there now if let's think about human driving it is not tabular assad learning so at the age of 16 in high school uh a teenager goes into uh goes into driver ed class right and now at that point they learn but at the age of 16 they are already visual geniuses because from 0 to 16 they have built a certain repertoire of vision in fact most of it has probably been achieved by age 2 right in in this period of age up to age 2 they know that the world is three-dimensional they know how objects look like from different perspectives they know about occlusion they know about common dynamics of humans and other bodies they have some notion of intuitive physics so they they built that up from their observations and interactions in early childhood and of course reinforced through their their growing up to age 16. so then at age 16 when they go into driver ed what are they learning they're not learning afresh the visual world they have a mastery of the visual world what they are learning is control okay they are learning how to be smooth about control about steering and brakes and so forth they're learning a sense of typical traffic situations now the the that education process can be quite short because they are coming in as visual geniuses and of course in their future they're going to encounter situations which are very novel right so during my driver ed class that i may not have had to deal with a skateboarder i may not have had to deal with a truck driving in front of me who's from who's where the back opens up and some junk gets dropped from the truck and i have to deal with it right but i can deal with this as a driver even though i did not encounter this in my driver at class and the reason i can deal with it is because i have all this general visual knowledge and expertise and uh do you think the learning mechanisms we have today can do that kind of long-term accumulation of knowledge or do we have to uh do some kind of you know in the the the work that led up to expert systems with knowledge representation you know the broader field of what of artificial intelligence uh worked on this kind of accumulation of knowledge do you think neural networks can do the same i think uh i don't see any in principle problem with neural networks doing it but i think the learning techniques would need to evolve significantly so the current uh the current learning techniques that we have yeah is our supervised learning you're given lots of examples xiy pairs and you you learn the functional mapping between them i think that human learning is far richer than that it includes many different components there are there is a a child explores the world and sees as for example a child takes an object and manipulates it in his or her hand and therefore gets to see the object from different points of view and the child has commanded the movement so that's a kind of learning data but the learning data has been arranged by the child and this is a very rich kind of data the child can do various experiments with the world so so there are many aspects of sort of human learning and these have been studied in in child development by psychologists and they what they tell us is that supervised learning is a very small part of it there are many different aspects of learning and what we would need to do is to develop models of all of these and then train our systems in that with that kind of uh protocol so new new methods of learning yes some of which might imitate the human brain but you also in your talks have mentioned some of the compute side of things the in terms of the difference in the human brain or referencing marvik hans marvel the so do you do you think there's something interesting valuable to consider about the difference in the computational power of the human brain versus the computers of today in terms of instructions per second yes so if we go back uh so so this is a point i've been making for 20 years now and i think once upon a time the way i used to argue this was that we just didn't have the computing power of the human brain our computers were uh were not quite there and i mean there is a well well-known trade-off which we know that the that neurons are slow compared to transistors but uh but we have a lot of them and they have a very high connectivity whereas in silicon you have much faster devices transistors switch at on the order of nanoseconds but the connectivity is usually smaller right at this point in time i mean we are now talking about 2020 we do have if you consider the latest gpus and so on amazing computing power and if we look back at enhanced modex type of calculations which he did in the 1990s we may be there today in terms of computing power comparable to the brain but it's not in the of the same style it's of a very different style so i mean for example the the style of computing that we have in our gpus is far far more power hungry than the style of computing that is there in the human brain or other biological uh entities yeah and that the efficiency part is uh we're gonna have to solve that in order to build actual real world systems of large scale let me ask sort of the high level question step taking a step back how would you articulate the general problem of computer vision does such a thing exist so if you look at the computer vision conferences and the work that's been going on it's often separated into different little segments breaking the problem of vision apart into whether segmentation 3d reconstruction object detection i don't know image capturing whatever uh there's benchmarks for each but if you were to sort of philosophically say what is the big problem of computer vision does such a thing exist yes but it's not in isolation so if we have to so for all intelligence tasks i always go back to sort of biology or humans and if we think about vision or perception in that setting we realize that perception is always to guide action perception in a for a biological system does not give any benefits unless it is coupled with action so we can go back and think about the first multicellular animals which arose in the cambrian era you know 500 million years ago and uh these animals could move and they could see in some ways and their two activities helped each other because uh uh how does movement help movement helps that because you can get food in different places but you need to know where to go and that's really about perception or seeing i mean i mean vision is perhaps the single most perception sense but all the others are equally are also important so uh so perception and action kind of grow go together so earlier it was in these very simple feedback loops which were about uh finding food or avoiding becoming food if there's a predator running uh trying to you know eat you up and and so forth so so we must at the fundamental level connect perception to action then as we evolved uh perception became more and more sophisticated because it served many more purposes and uh so today we have what seems like a fairly general purpose capability which can look at the external world and build and a model of the external world inside the head we do have that capability that model is not perfect and psychologists have great fun in pointing out the ways in which the model in your head is not a perfect model of the external world and they have create various illusions to show the ways in which it is imperfect but it's amazing how far it has come from a very simple perception action loop that you exists in you know an animal 500 million years ago once we have this these very sophisticated visual systems we can then impose a structure on them it's we as scientists who are imposing that structure where we have chosen to characterize this part of the system as this code module of object detection or quote this module of 3d reconstruction what's going on is really all of these processes are running simultaneously and uh and and they are running simultaneously because originally their purpose was in fact to help guide action so as a guiding general statement of a problem do you think we can say that the the general problem of computer vision you said in humans it was tied to action do you think we should also say that ultimately the the goal the problem of computer vision is to sense the world in the way that helps you act in the world yes i think that's the most fundamental uh that's the most fundamental purpose we have by now hyper evolved so we have this visual system which can be used for other things for example judging the aesthetic value of a painting and this is not guiding action maybe it's guiding action in terms of how much money you will put in your auction bid but that's a bit stretched but the basics are in fact in terms of action but we have we've evolved really this hyper uh we have hyper evolved our visual system actually just too uh sorry to interrupt but perhaps it is fundamentally about action you kind of jokingly said about spending but perhaps the capitalistic uh drive that drives a lot of the development in this world is is about to exchange your money and the fundamental action is money if you watch netflix if you enjoy watching movies you're using your perception system to interpret the movie ultimately your enjoyment of that movie means you'll subscribe to netflix so the action is this uh this extra layer that we've developed in modern society perhaps this is fundamentally tied to the action of spending money well certainly with respect to uh you know interactions with firms so so in this homo economics role when you're interacting with firms it does become uh it does become that that's what else is there uh that was a rhetorical question okay so to to linger on the division between the static and the dynamic so much of the work in computer vision so many of the breakthroughs that you've been a part of have been in the static world in looking at static images and then you've also worked on starting but it's a much smaller degree the community is looking at dynamic and video at dynamic scenes and then there is robotic vision which is dynamic but also where you actually have a robot in the physical world interacting based on that vision which problem is harder the the the intuit sort of the the trivial first answers well of course one image is harder but so if you look at a deeper question there are we um what's the term cutting ourselves cutting ourselves at the knees or like making the problem harder by focusing on the images that's a fair question i think sometimes we we can simplify our problem so much that we essentially lose part of the juice that could enable us to solve the problem and one could reasonably argue that to some extent this happens when we go from video to single images now historically uh you have to consider the limits of imposed by the competition capabilities we had so if we many of the choices made in the computer vision community uh through the 70s 80s 90s can be understood as choices which were forced upon us by the fact that we just didn't have access to compute enough compute not enough memory none of hard drives not exactly not enough not enough compute not enough storage so so think of these choices so one of the choices is focusing on single images rather than video okay clear questions storage and compute we had to focus on we did we used to detect edges and throw away the image right so you have an image which i say 256 by 256 pixels and instead of keeping around the grayscale value what we did was we detected edges find the places where the brightness changes a lot so now that and now and then throw away the rest so this was a major compression device and the hope was that this makes it that you can still work with it and the logic was humans can interpret a line drawing and uh and yes and this will save us a competition so many of the choices were dictated by that i think uh today we are no longer detecting edges right we process images with convnets because we don't need to we don't have that those compute restrictions anymore now video is still under studied because video compute is still quite challenging if you are a university researcher i think video computing is not so challenging if you are at google or facebook or amazon still super challenging i've just spoke with the vp of engineering google head of the youtube search and discovery and they still struggle doing stuff on video it's very difficult except doing except using techniques that are essentially the techniques you used in in the 90s some very basic computer vision techniques no that's when you want to do things at scale so if you want to operate at the scale of all the content of youtube it's very challenging and there's similar issues in facebook but as a researcher you you have you have more uh you know opportunities you can train large you know that works with relatively large uh video data sets yeah yes so i think that this is part of the reason why we have so emphasized static images i think that this is changing and over the next few years i see a lot more progress happening in in video so i have this generic statement that to me video recognition feels like 10 years behind object recognition and you can quantify that because you can take some of the challenging video data sets and their performance on action classification is like say 30 which is kind of what we used to have around 2009 in object detection you know so it's like about 10 years behind and uh whether it'll take 10 years to catch up is a different question hopefully it will take less than that let me ask a similar question i've already asked but once again so for dynamic scenes do you think do you think some kind of injection of knowledge basis and reasoning is required to help improve like action recognition like if if if um if we solve the general action recognition problem what do you think the solution would look like it's another way yeah so i i completely agree that knowledge is called for and that knowledge can be quite sophisticated so the way i would say it is that perception blends into cognition and cognition brings in issues of memory and this notion of a schema from psychology which is uh let me use the classic example which is you go to a restaurant right now the things that happen in a certain order you walk in somebody takes you to a table a waiter comes gives you a menu takes the order food arrives eventually a bill arrives etc etc this is a classic example of ai from the 1970s uh it was called there was the term frames and scripts and schemas these are all quite similar ideas okay in the 70s the way the ai of the time dealt with it was by build hand coding this so they hand coded in this notion of a script and the various stages and the actors and so on and so forth and use that to interpret for example language i mean if there's a description of a of a story involving some people eating at a restaurant there are way all these inferences you can make because you know what happens typically at a restaurant so i think this kind of uh this kind of knowledge is absolutely essential so i think that when we are going to do long-form video understanding we are going to need to do this i think the kinds of technology that we have right now with 3d convolutions over a couple of seconds of clip or video it's very much tailored towards short-term video understanding not that long-term understanding long-term understanding requires a notion of this notion of schemas that i talked about perhaps some notions of goals intentionality functionality and so on and so forth now how will we bring that in so we could either revert back to the 70s and say okay i'm going to hand code in a script or we might try to learn it so i tend to believe that we have to find learning ways of doing this because i think learning ways to land up being more robust and there must be a learning version of the story because uh children acquire a lot of this knowledge by uh sort of just observation so at no moment in a child's life there's a it's possible but i think it's not so typical that somebody that a mother coaches a child through all the stages of what happens in a restaurant they just go as a family they they they go to the restaurant they eat come back and the child goes through 10 such experiences and the child has has got a schema of what happens when you go to a restaurant so we somehow need to we need to provide that capability to our systems you mentioned the following line from the end of the alan turing paper uh computing machinery and intelligence that many people like you said many people know and very few have read where he proposes the turing test this is this is how you know because it's towards the end of the paper instead of trying to produce a program to simulate the adult mind why not rather try to produce one which simulates the child's so that's a really interesting point if i think about the benchmarks we have before us the the tests of our computer vision systems they're often kind of trying to get to the adult so what kind of benchmarks should we have what kind of tests for computer vision do you think we should have that mimic the child's in computer vision yeah i think we should have those and we don't have those today and i think uh the part of that the challenge is that we should really be collecting data of the type that a child uh that the child experiences right so that gets into issues of you know privacy and so on and so forth but there are attempts in this direction to sort of try to collect the kind of data that a child encounters growing up so what's the child's linguistic environment what's the child's visual environment so if we could collect that kind of data and then develop learning schemes based on that data that would be one way to do it i i think that's a very promising direction myself there might be people who would argue that we could just short circuit this in some way and uh sometimes we have imitated uh we have not we have had success by not imitating nature in detail so the usual example is airplanes right we don't build flapping winds flapping wings so uh yes that's uh that's one of the points of debate uh in my mind i i i would i would bet on this this learning like a child approach so one of the fundamental aspects of learning like a child is the interactivity so the child gets to play with the data set it's learning from yes it's against the select i mean you can call that active learning you can you know in the machine learning world you can call it a lot of terms what are your thoughts about this whole space of being able to play with the data set or select what you're learning yeah so i think that uh i i believe in that and i think that we could achieve it in in two ways and i think we should use both so one is uh actually real robotics right so real uh you know physical embodiments of agents who are interacting with the world and they have a physical body with dynamics and mass and moment of inertia and friction and all the rest and you learn your body the robot learns its body by doing a series of actions the second is that simulation environments so i think simulation environments are getting much much better in my in my life in facebook ai research our group has worked on something called habitat which is a simulation environment which is a visually photorealistic environment of you know places like houses or interiors of various urban spaces and so forth and as you move you get a picture which is a pretty accurate picture so uh i i can now uh you can imagine that subsequent generations of these simulators will be accurate not just visually but with respect to you know forces and masses and haptic interactions and so on and uh then then we have that environment to play with i think that let me state one reason why i think this active being able to act in the world is important i think that this is one way to break the correlation versus causation barrier so this is something which is of a great deal of interest these days i mean people like judea pearl have talked a lot about uh why that we are neglecting causality and he describes the entire set of successes of deep learning as just curve fitting right because it's uh but i i don't quite agree about as a troublemaker he is but uh causality is important but causality is not is not like a single silver bullet it's not like one single principle there are many different aspects here and one of the ways in which uh one of our most reliable ways of establishing causal links and this is the way for example the the medical community does this is randomized control trials so you have you you pick some situation and now in some situation you perform an action and for certain others you don't right so so you have a control experiment well the child is in fact performing controlled experiments all the time right right right okay small scale and in a small scale and but but that is a way that the child gets to build and refine its causal models of the world and my colleague alison gopnik has together with a couple of authors co-authors has this book called the scientist in the crib referring to children so i like the part that i like about that is the scientist wants to do wants to build causal models and the scientist does control experiments and i think the child is doing that so to enable that we will need to have these these active experiments and i think this could be done some in the real world and some in simulation so you have hope for simulation i have a hopeless solution that's an exciting possibility if we can get to not just photo realistic but what's that called life realistic yeah uh simulation so you don't see any fundamental blocks to why we can't eventually simulate the the principles of what it means to exist in the world as a physical i i don't see any fundamental problems there i mean and look the computer graphics community has come a long way right so the in the early days back going back to the 80s and 90s they were they were focusing on visual realism right and then they could do the easy stuff but they couldn't do stuff like hair or fur and so on okay well they managed to do that then they couldn't do physical actions right like there's a bowl of glass and it falls down and it shatters but then they could start to do pretty realistic models of that and so on and so forth so the graphics people have shown that they can do this forward direction not just for optical interactions but also for physical interactions so i think uh of course some of that is very computer intensive but i think by and by we will find ways of making our models ever more realistic you break vision apart into in one of your presentations early vision static scene understanding dynamics and understanding and raise a few interesting questions i thought i could just throw some some at you just to see if you want to talk about them so early vision so it's what is it you said um sensation perception and cognition so is this a sensation yes what can we learn from image statistics that we don't already know so at the lowest level what um what can we make from just this the the statistic the basics so there were the variations in the rock pixels the textures and so on yeah so what we seem to have learned is uh uh uh is that there's a lot of redundancy in these images and as a result we are able to do a lot of compression and and this compression is very important in biological settings right so you might have ten to the eight photoreceptors and only ten to the six fibers in the optic nerve so you have to do this compression by a factor of hundreds to one and uh and uh so there are analogs of that which are happening in in our neural net artificial neural network that's the early layer so you think there's a lot of compression that can be done in the beginning yeah just just the statistics yeah um how much how much well so i mean the the way to think about it is just how successful is image compression right and we we and there are and that's been done with older technologies but it can be done with there are several companies which are trying to use sort of these more advanced neural network type techniques for compression both for static images as well as for for video one of my former students has a company which is trying to do stuff like this and i think i think that they are showing quite interesting results and i think that that's all the success of that's really about image statistics and video statistics but that's still not doing compression of the kind when i see a picture of a cat all i have to say is it's a cat that's another semantic kind of complication yeah so this is this is at the lower level right so we are we are we as i said yeah that's focusing on low level statistics so to linger on that for a little bit uh you mentioned how far can bottom-up image segmentation go and in general what you mentioned that the central question for scene understanding is the interplay of bottom-up and top-down information maybe this is a good time to elaborate on that maybe define what is what is up what is top down in the comments yes the computer vision uh right that's uh so today what we have are a are very interesting systems because they work completely bottom up how are they what does bottom bottom-up mean sorry so bottom-up means in this case means a feed-forward net neural network so starting from the raw pixels yeah they start from the raw pixels and they they end up with some something like cat or not a cat right so our our systems are running totally feed forward they're trained in a very top-down way so they're trained by saying okay this is a cat there's a cat there's a dog there's a zebra etc and i'm not happy with either of these choices fully we have gone into uh because we have completely separated these processes right so there is a so i would like the uh the process uh so what do we know compared to biology so in biology what we know is that the processes in at test time at run time those processes are not purely feed forward but they involve feedback so and they involve much shallower neural networks so the kinds of neural networks we are using in computer vision say a resnet 50 has 50 layers well in in the brain in the visual cortex going from the retina to it maybe we have like seven right so they're far shallower but we have the possibility of feedback so there are backward connections and this might enable us to uh to deal with the more ambiguous stimuli for example so the the biological solution seems to involve feedback the solution in in artificial vision seems to be just feed forward but with a much deeper network and the two are functionally equivalent because if you have a feedback network which just has like three rounds of feedback you can just unroll it and make it three times the depth and create it in a totally feed forward way so this is something which i mean we have written some papers on this theme but i really feel that this should this theme should be pursued further have some kind of recurrence mechanism yeah okay the other uh so that so that's uh so i so i want to have a little bit more top down in the at test time okay then at training time we make use of a lot of top-down knowledge right now so basically to learn to segment an object we have to have all these examples of this is the boundary of a cat and this is the boundary of a chair and this is the boundary of a horse and so on and this is too much top-down knowledge how do humans do this we manage to we manage with far less supervision and we do it in a sort of bottom-up way because for example we're looking at a video stream and the horse moves and that enables me to say that all these pixels are together yeah so the gestural psychologists used to call this the principle of common fate so there was a bottom-up process by which we were able to segment out these objects and we have totally focused on this top-down training signal so in my view we have currently solved it in machine vision this top-down bottom-up interaction but i don't find the solution fully satisfactory and i would rather have a bit of both in at both stages for all computer vision problems which is not just segmentation and and and and the question that you can ask is so for me i'm inspired a lot by human vision and i care about that you could be a just a hard-boiled engineer not give a damn so to you i would then argue that uh you would need far less training data if you could make my uh research agenda you know fruitful okay so maybe taking a step into uh segmentation static scene understanding what is the interaction between segmentation and recognition you mentioned the movement of objects so for people who don't know computer vision segmentation is this weird activity that we that computer vision folks have all agreed is very important uh of drawing outlines around objects versus a bounding box or and then classifying that object what's what's the value of segmentation what is it as a problem in computer vision how is it fundamentally different from detection recognition any other problems yeah so i think uh so so segmentation enables us to say that some set of pixels are an object without necessarily even being able to name that object or knowing properties of that object oh so you mean segmentation purely as as as the act of separating an object from its background a blob of uh of that's united in some way from his background yeah so identification if you were making an entity out of it and justification yeah beautifully so so i think that we have that capability and that is that enables us to uh as we are growing up to acquire uh names of objects with very little supervision so suppose the child lets posit that the child has this ability to separate out objects in the world then when the there's a the mother says pick up your bottle or the cat's behaving funny today [Laughter] the word cat suggests some object and then the child sort of does the mapping right right the mother doesn't have to teach a specific object labels by pointing to them weak supervision works in the context that you have the ability to create objects so i think that uh so to me that's that's a very fundamental capability uh there are applications where this is very important uh for example medical diagnosis so in medical diagnosis uh you have some uh brain scan i mean some this is some work that we did in my group where you have ct scans of people who have had traumatic brain injury and what uh what the radiologist needs to do is to precisely delineate various places where there might be bleeds for example and there's there are clear needs like that so they're certainly very practical applications of computer vision where segmentation is necessary but philosophically segmentation enables the task of recognition to proceed with much weaker supervision than we require today and you think of segmentation as this kind of task that takes on a visual scene and breaks it apart into into interesting entities yeah that might be useful for whatever the task is yeah and and it is not semantics free so i think i i mean it it blends into it involves perception and cognition it is not it is not i i think the mistake that we used to make in the early days of computer vision was to treat it as a purely bottom-up perceptual task it is not just that because we do revise our notion of segmentation with more experience right because for example there are objects which are non-rigid like animals or humans and uh i think understanding that all the pixels of a human are one entity is actually quite a challenge because the parts of the human they can move independently and the human wears clothes so they might be differently colored so it's all sort of a challenge you mentioned the three hours of computer vision are recognition reconstruction reorganization can you describe these three r's sure how they interact yeah so uh so recognition is the easiest one because that's uh what i think people generally think of as computer vision achieving these days which is uh labels so is this a cat is this a dog is this a chihuahua i mean you know it could be very fine grain like you know specific breed of a dog or a specific species or bird or it could be very abstract like animal but given a part of an image or a whole image say put a label on that yeah so that's that's recognition reconstruction is uh essentially it you can think of it as inverse graphics i mean that's one way to think about it so graphics is your you have some internal computer representation and uh you have a computer representation of some objects arranged in a scene and what you do is you produce a picture you produce the pixels corresponding to a rendering of that scene so uh so let's do the inverse of this we are given an image and we try to we we we say oh this image arises from some objects in a scene looked at with a camera from this viewpoint and we might have more information about the objects like their shape maybe their textures maybe you know color et cetera et cetera so that's the reconstruction problem in a way that you are in your head creating a model of the external world okay reorganization is to do with essentially finding these entities so uh so it's uh organization or the word organization implies structure so uh that in in uh perception in psychology we use the term perceptual organization that uh the the world is not just an image is not just seen as is not internally represented as just a collection of pixels but we make these entities we create these entities objects whatever you want to call in the relationship between the entities as well or is it purely about the entities it could be about the relationships but mainly we focus on the fact that there are entities sometimes i'm trying to pinpoint what the organization means so organization is that instead of like a uniform grid we have the structure of objects so segmentation is a small part of that so segmentation gets us going towards that yeah and you kind of have this triangle where they all interact together yes so how do you see that interaction in uh sort of uh reorganization is yes defining the entities in the world the recognition is labeling those entities and then reconstruction is what filling in the gaps well to for example see impute some 3d objects corresponding to each of these entities that would be part of adding more information that's not there in the raw data correct i mean i started pushing this kind of a view in the around 2010 or something like that because at that time in computer vision the distinction that people were were just working on many different problems but they treated each of them as a separate isolated problem with each with its own data set and then you try to solve that and get good numbers on it so i wasn't i didn't like that approach because i wanted to see the connection between these and if people divided up vision into into various modules the way they would do it is as low level mid-level and high-level vision corresponding roughly to the psychologist's notion of sensation perception and cognition and i didn't that didn't map to tasks that people cared about okay so therefore i tried to promote this particular framework as a way of considering the problems that people in computer vision were actually working on and trying to be more explicit about the fact that they actually are connected to each other and i was at that time just doing this on the basis of information flow now it turns out in the last five years or so in the post the deep learning revolution that this this architecture has turned out to be very conducive to that because basically in these neural networks we are trying to build multiple representations there can be multiple output heads sharing common representations so in a certain sense today given the reality of what solutions people have to these i i i i do not need to preach this anymore it is it is just there it's part of the solution space so speaking of neural networks how much of this uh problem of computer vision of the organization recognition can be um reconstruction how much of it can be learned end to end do you think instead of uh set it and forget it just plug and play have a giant data set multiple perhaps multi-modal and then just learn the entirety of it well so i i think that currently what that end-to-end learning means nowadays is end-to-end supervised learning and and that i would argue is too narrow a view of the problem i would i like this child development view this lifelong learning view one where there are certain capabilities that are built up and then there are certain capabilities which are built up on top of that so uh that's that's what i i believe in so i think uh end-to-end learning in the supervised setting for a very precise task to me is a kind of is uh it's sort of a limited view of the of the learning process got it so if we think about beyond purely supervised look at back to children you mentioned six lessons that we can learn from children uh of be multimodal be incremental be physical explore be social use language can you speak to these perhaps picking one that you find most fundamental toward yeah time today yeah so i mean i should say to give due credit this is from a paper by smith and gasser and it reflects essentially i would say common wisdom among child development people it's just that these are this is not common wisdom among people in computer vision and ai and machine learning so i view my role as uh trying to bridge the worlds bridge the two worlds so uh so let's take an example of a multi-modal i like that so multi-modal canonical example is uh a child interacting with uh with an object so then the child so the child holds a ball and plays with it so at that point it's getting a touch signal so the touch signal is is getting as the notion of 3d shape but it is sparse and then the child is also seeing a visual signal right and and these two so imagine these are two in totally different spaces right so one is the space of receptors on the skin of the fingers and the thumb and the palm right and then these map on to these neuronal fibers are getting activated somewhere right these lead to some activation in somatosensory cortex i mean a similar thing will happen if we have a robot hand okay and then we have the pixels corresponding to the visual view but we know that they correspond to the same object right so that's a very very strong cross calibration signal and it is self-supervisory which is beautiful right there's nobody assigning a label the mother doesn't have to come and assign a label the child doesn't even have to know that this object is called a ball okay but the obj the child is learning something about the three-dimensional world from this signal uh i think tactile and visual there is some work on there is a lot of work currently on audio and visual okay an audio visual so there is some event that happens in the world and that event has a visual signature and it has a auditory signature so there is this glass bowl on the table and it falls and breaks and i hear the smashing sound and i see the pieces of glass okay i've built that connection between the two right we have people uh i mean this has become a hot topic in computer vision in the last couple of years there is there are problems like uh separating out multiple speakers right which was a classic problem in in audition they call this the problem of source separation or the cocktail party effect and so on but just try to do it visually when you also have it becomes so much easier and so much more useful so the the multimodal i mean there's so much more signal with multimodal and you can use that for some kind of weak supervision as well yes because they are occurring at the same time in time yeah so you have time which links the two right so at a certain moment t1 you've got a certain signal in the auditory domain and a certain signal in the visual domain but they must be causally related yeah it's an exciting area not well studied yet not yeah i mean we have a little bit of work at this but uh but but so much more needs to be done yeah so so so so this this is this is a good example be physical that's to do with uh like the one thing we talked about earlier that that there's a embodied world to mention language use language so no chomsky believes that language may be at the core of cognition at the core of everything in the human mind what is the connection between language and vision to you like what's more fundamental are they neighbors is one the parent and the child the chicken and the egg oh it's very clear it is vision which is the appearance the fundament the permission is the fundamental ability okay well so uh it comes before you think vision is more fundamental than language correct and and and it and yeah you can think of it either in phylogeny or in ontogeny so phylogeny means if you look at evolutionary time right so you we have vision that developed 500 million years ago okay then something like when we get to maybe like five million years ago you have the first bipedal primate so when we started to walk then the hands became free and so then manipulation the ability to manipulate objects and build tools and so on and so forth so you said 500 000 years ago no no sorry the the first multicellular animals which you can say had some intelligence arose 500 million years ago okay and now let's fast forward to say the last seven million years which is the development of the hominid line right where from the other primates we have the branch which leads on to modern humans now there are many of these hominids but the the ones which you know people talk about lucy because that's like a skeleton from three million years ago and we know that lucy walked okay so at this stage you have that the hand is free for manipulating objects and then the ability to manipulate objects build tools and the brain size grew in this era so okay so now you have manipulation now we don't know exactly when language arrows but after that but after that because no apes have i mean so i mean chomsky is correct in that that it is a uniquely human capability and we primates other primaries don't have that but so it developed somewhere in this era but it developed i would i mean uh argue that it probably developed after we had this stage of uh uh humans or i mean the human species already able to manipulate and a hands-free much bigger brain size and for that there's a lot of vision has already had had to have developed yeah so the sensation and the perception may be some of the cognition yeah so we we so those so so that so the world so there so so these ancestors of us you know three four million years ago they had uh they had spatial intelligence so they knew that the world consists of objects they knew that the objects were in certain relationships to each other they had observed causal interactions among objects they could move in space so they had space and time and all of that so language builds on that substrate so language has a lot of i mean i mean the all human languages have constructs which depend on a notion of space and time where did that notion of space and time come from it had to come from perception and action in the world we live in yeah what you refer to as the spatial intelligence yeah yeah to linger a little bit we mentioned touring and his uh mention of we should learn from children nevertheless language is the fundamental piece of the test of intelligence that touring proposed what do you think is a good test of intelligence are you what would impress the heck out of you is it fundamentally natural language or is there something in vision i i think uh i i wouldn't i i don't think we should have created a single test of intelligence so just like i don't believe in iq as a single number i think generally there can be many capabilities which are correlated perhaps so i think that there will be uh there will be accomplishments which are visual accomplishments accomplishments which are uh accomplishments in manipulation or robotics and then accomplishments in language i do believe that language will be the hardest not to crack really yeah so what's what's harder to pass the spirit of the touring test or like whatever formulation will make it natural language convincingly in natural language like somebody you would want to have a beer with hang out and have a chat with or the general natural scene understanding you think language is the type i think i'm not a fan of the i think i think turing test that turing as he proposed the test in 1950 was trying to solve a certain problem yeah imitation yeah and and i think it made a lot of sense then where we are today 70 years later i think i think we we should not worry about that i mean i think the turing test is no longer the right way to uh to to channel research in in ai because that it takes us down this path of this chat bot which can fool us for five minutes or whatever okay i think i would rather have a list of 10 different tasks i mean i think their tasks which their tasks in the manipulation domain tasks and navigation tasks and visual scene understanding tasks in under reading a story and answering questions based on that i mean so my favorite language understanding task would be you know reading a novel and being able to answer arbitrary questions from it okay right i i think that to me uh and this is not an exhausted list by any means so i would uh i think that that's what we where we need to be going to and each of these on each of these axes there's a fair amount of work to be done so on the visual understanding side in this intelligence olympics that we've set up yeah what's a good test for one of many of visual scene understanding uh do you think such benchmarks exist sorry to interrupt no there there aren't any i i think i think essentially to me a really uh good aid to the blind so suppose there was a blind person and i needed to assist the blind person so ultimately like we said vision that aids in the action in the survival in this world yeah maybe in a simulated world maybe easier to to measure performance in a simulated world what we are ultimately after is performance in the real world so david hilbert in 1900 proposed 23 open problems in mathematics some of which are still unsolved most important famous of which is probably the riemann hypothesis you've thought about and presented about the hilbert problems of computer vision so let me ask what to you today i don't know when the last year you presented that 2015 but versions of it yeah you're kind of the the face and the spokesperson for computer vision yeah it's your job to just to state what the problem the open problems are for the field so what today are the hilbert problems of computer vision do you think let me pick pick one to which i regard as uh clearly clearly unsolved which is what i would call long-form video understanding so so we have a video clip and we want to understand the behavior in there in terms of agents their goals intentionality and uh make predictions about what might happen you know so so that that kind of understanding which goes away from atomic visual action so so in the short range the question is are you sitting are you standing are you catching a ball right that we can do now or we even if we can't do it fully accurately if we can do it at 50 percent maybe next year we'll do it at 65 and so forth but i think the long range video understanding i don't think we we we can do today well today and that means so long and it blends into cognition that's the reason why it's challenging and so you have to track you have to understand the entities you have to understand the sds you have to track them and you have to have some kind of model of their behavior correct and their and if their behavior might be these are these are agents so they are not just like passive objects but the agent so therefore we they might they would exhibit gold directed behavior okay so this is this is one area then i will talk about say understanding the world in 3d now this may seem paradoxical because in a way we have been able to do 3d understanding even like 30 years ago right but i don't think we currently have the richness of 3d understanding in our computer vision system that we would like because ah so let me elaborate on that a bit so currently we have two kinds of techniques which are not fully unified so there are the kinds of techniques from multi-view geometry that you have multiple pictures of a scene and you do a reconstruction using stereoscopic vision or structure from motion but these techniques do not they totally fail if you just have a single view because they are relying on this this multiple geometry okay then we have some techniques that we have developed in the computer vision community which try to guess 3d from single views and these techniques are based on on supervised learning and they are based on having a training time 3d models of objects available and this is completely unnatural supervision right that's not cad models are not injected into your brain okay so what would i like what i would like would be a kind of uh learning as you move around the world uh notion of 3d so so we we have our succession of visual experiences and from those we so in as part of that i might see a chair from different viewpoints or a table from viewpoint different viewpoints and so on now as part that enables me to build some internal representation and then next time i just see a single photograph and it may not even be of that chair it's of some other chair and i have a guess of what its 3d shape is like so you're almost learning the cad model kind of yeah implicitly i mean implicitly i mean the cad model need not be in the same form as used by computer graphics hidden in the representation it's hidden in the representation the ability to predict new views and what i would see if i went to such and such position by the way and on a small tangent on that are you uncomforta are you okay or comfortable with neural networks that do achieve visual understanding that do for example achieve this kind of 3d understanding and you don't know how they you don't know the rep you're not able to interest but you're not able to visualize or understand or interact with the representation so the fact that they're not or may not be explainable yeah i think that's fine i to me that is uh so so let me put some caveats on that so it depends on the setting so first of all i think uh uh the uh humans are not explainable so yeah that's a really good point yeah so we we one human to another human is not fully explainable i think there are settings where explainability matters and these might these are these might be for example questions on medical diagnosis so i'm in a setting where maybe the doctor maybe a computer program has made a certain diagnosis and then depending on the diagnosis perhaps i should have treatment day or treatment b right so now is the computer programs diagnosis based on data which was data collected of for american males who are in their 30s and 40s and maybe not so relevant to me maybe it is relevant you know et cetera et cetera and we i mean in medical diagnosis we have major issues to do with the reference class so we may have acquired statistics from one group of people and applying it to a different group of people who may not share all the same characteristics the data might have there might be error bars in the prediction so that prediction should really be taken with a huge grain of salt and but this has an impact on what treatments should be picked right so so there are settings where i want to know more than just this is the answer but what i acknowledge is that so so so so i in that sense explainability and interpretability may matter it's about giving error bounds and a better sense of the quality of the decision where what i where i'm willing to sacrifice interpretability is that i believe that there can be systems which can be highly performant but which are internally black boxes and and that seems to be words headed some of the best performing systems are essentially black boxes yeah uh fundamentally by their construction you and i are black boxes to each other yeah so the nice thing about the black boxes we are is so we ourselves are black boxes but we're also those of us who are charming are able to convince others like explain the black what's going on inside the black box with narratives with stories so in some sense uh neural networks don't have to actually explain what's going on inside they just have to come up with stories real or fake that convince you that they know what's going on and i'm sure we can do that we can create those nearer those stories neural networks can create those stories yeah and the transformer will be involved do you think we will ever build a system of human level or superhuman level intelligence we've kind of defined what it takes to try to approach that but do you think we'll do you think that's within our reach the thing that we thought we could do what touring thought actually we could do by a year 2000 right what do you think we'll ever be able to do so i think there are two answers here one question one answer is in principle can we do this at some time and my answer is yes the second answer is a pragmatic one do you think we will be able to do it in the next 20 years or whatever and to that man says no so and of course that's a wild guess i i i i think that you know donald trump's felt is not a favorite person of mine but one of his lines is very good which is about known knowns known unknowns and unknown unknowns so in the business we are in there are known unknowns and we have unknown unknowns so i think with respect to a lot of what the case in vision and robotics i feel like we have known unknowns so i have a sense of where we need to go and what the problems that need to be solved are i feel with respect to natural language understanding and high level cognition it's not just known unknowns but also unknown unknowns so it is very difficult to put any kind of uh time frame to that uh do you think some of the unknown unknowns might be positive in that they'll surprise us and make the job much easier so fundamental breakthroughs i think that is possible because certainly i have been very positively surprised by how effective these deep learning systems have been because i certainly would not have believed that in 2010 i think what we knew from the mathematical theory was that convex optimization works when there's a single global optima then these gradient descent techniques would work now these are non-linear systems with non-convex systems huge number of variables so over-parametrized over-parameterized and the people who used to play with them a lot the ones who are totally immersed in the lore and the black magic they knew that they worked uh well even though they were really i thought like everybody no the claim that i hear from my friends like yan lacoon and so forth now yeah that they feel that they were comfortable with them well he says but the community as a whole was certainly not and i think uh we were to me that was the surprise that they actually worked robustly for a wide range of problems from a wide range of initializations and so on and uh so that was that that was certainly more rapid progress than uh we expected but then there are certainly lots of times in fact most of the history and fear is when we have made less pro progress at a slower rate than we expected so uh we just keep going i think uh what i regard as uh really unwarranted are these these fears of uh you know agi in 10 years and 20 years and that kind of stuff because that's based on completely unrealistic models of how rapidly we will make progress in this field so i agree with you but i've also gotten a chance to interact with very smart people who really worry about the existential threats of ai and i as an open-minded person and sort of taking and taking it in do you think if ai systems in some way the unknown unknowns not super intelligent ai but in ways we don't quite understand uh the nature of superintelligence will have a detrimental effect on society do you think this is something we should be worried about or we need to first allow the unknown our nose to become known unknowns i think we need to be worried about ai today i think that it is not just a worry we need to have when we get that agi i think that ai is being used in many systems today and there might be settings for example when it causes biases or decisions which could be harmful i mean decisions which could be unfair to some people or it could be a self-driving cars which kills a pedestrian so ai systems are being deployed today right and they're being deployed in many different settings maybe in medical diagnosis maybe in a self-driving car maybe in selecting applicants for an interview so i would argue that when these systems make mistakes there are consequences and we are in a certain sense responsible for those consequences so i would argue that this is a continuous effort it is we and and this is something that in a way is not so surprising it's about all engineering and scientific progress which uh great power comes great responsibility so as these systems are deployed we have to worry about them and it's a continuous problem i don't think of it as something which will suddenly happen on some day in 2079 for which i need to design some clever trick i'm saying that these problems exist today yeah and we need to be continuously on the lookout for worrying about safety biases risks right i mean the self-driving car kills are pedestrian and they have right i mean the this uber incident in arizona yeah right it has happened right this is not about agi it in fact it's about a very dumb intelligence which is also killing people the worry people have with agi is the scale and i but i think you're 100 right is like the thing that worries me about ai today and it's happening in a huge skills recommend recommender systems recommendation systems so if you look at twitter or facebook or youtube their controlling the ideas that we have access to the news and so on and that's a fundamentally machine learning algorithm behind each of these recommendations and they i mean my life would not be the same without these sources of information i'm a totally new human being and the ideas that i know are very much because of the internet because of the algorithm that i recommend those ideas and so as they get smarter and smarter i mean that is the agi yeah is that's the the algorithm that's recommending the next youtube video you should watch has control of millions of billions of people that that algorithm is already super intelligent and has complete control of the population not a complete but very strong control for now we can turn off youtube we can just go have a normal life outside of that but the more and more that gets into our life it's that algorithm we start depending on it in the different companies that are working on the algorithm so i think it's you're right it's already it's already there and youtube in particular is using computer vision doing their hardest to try to understand the content of videos so they could be able to connect videos with the people who would benefit from those videos the most and so that development could go in a bunch of different directions some of which might be harmful so yeah you're right the the the threats of ai are here already we should be thinking about them on a philosophical notion if you could personal perhaps if you could relive a moment in your life outside of family because it made you truly happy or was a profound moment that impacted the direction of your life what would you go to i don't think of single moments but i look over the long haul i feel that i've been very lucky because i feel that i think that in scientific research a lot of it is about being at the right place at the right time and you can you can work on problems at a time when they're just too premature you know you butt your head against them and and nothing happens because it's the prerequisites for success are not there and then there are times when you are in a field which is all pretty mature and you can only solve curricules upon colloquius i've been lucky to have been in this field which for 34 years 35 well actually 34 years as a professor at berkeley so longer than that uh which when i started in it was just like some little crazy absolutely useless field which couldn't really do anything to a time when it's really really solving a lot of practical problems has a lot has offered a lot of tools for scientific research right because computer vision is impactful for images in biology or astronomy and and so on and so forth and we have so we have made great scientific progress which has had real practical impact in the world and i feel lucky that i i got in at a time when the field was very young and at a time when it is it's now mature but not fully mature it's mature but not done i mean it's really in still in a in a productive phase yes yeah yeah i think people 500 years from now would laugh are you calling this field mature yeah that is very possible yeah so but you're also lest i forget to mention you've also mentored some of the biggest names of computer vision computer science and ai today uh there's so many questions i could ask but really is what what is it how did you do it what does it take to be a good mentor what does it take to be a good guide yeah i i think what i feel i've been lucky to have had very very smart and hardworking and creative students i think some part of the credit just belongs to being at berkeley i think those of us who are at top universities are blessed because we have very very smart and capable students coming on knocking on our door so so i have to be humble enough to acknowledge that but what have i added i think i have added something what i have added is uh i think what i've always tried to teach them is a sense of picking the right problems so i think that in science in the short run success is always based on technical competence your you know you're quick with math or you are whatever i mean there's certain technical capabilities which make for short-range progress long-range progress is really determined by asking the right questions and focusing on the right problems and i feel that what i've been able to bring to the table in terms of advising these students is some sense of taste of what are good problems what are problems that are worth attacking now as opposed to waiting 10 years what's a good problem if you could summarize if is that possible to even summarize like what what's your sense of a good problem i i think uh i think uh i have a sense of what is a good problem which is uh there is a british scientist uh in fact he won a nobel prize peter medover who has a a book on on this and uh basically he calls it the research is the art of the soluble so we need to sort of find problems which are which are not yet solved but which are approachable and he sort of refers to this sense that there is this problem which isn't quite solved yet but it has a soft underbelly there is some place where you can you know spear the beast yes and having that intuition that this problem is ripe is is a good thing because otherwise you can just beat your head and not make progress so i think that is that is important so if if i have that and if i can convey that to students it's not just that they do great research while they're working with me but that they continue to do great research so in a sense i'm proud of my students and their achievements and their great research even 20 years after they've seized being my student so it's in part developing helping them develop that sense that a problem is not yet solved but it's solvable correct the other thing which i have which i i think i bring to the table uh is i is a certain intellectual breadth i i've spent a fair amount of time studying psychology neuroscience relevant areas of applied math and so forth so i can probably help them see some connections to disparate things which they might not have otherwise so so the smart students coming into berkeley can be very uh deep in the sense they can think very deeply meaning very hard down one particular path but where i could help them is the the shallow breadth but uh whereas they would have the the narrow depth and uh but that's that's of some value well it was beautifully refreshing just to hear you naturally jump to psychology back to computer science and this conversation back and forth i mean that that's uh that's actually a rare quality and i think it's certainly for students empowering to think about problems in a new way so for that and for many other reasons i really enjoyed this conversation thank you so much it was a huge honor thanks for talking today it's been my pleasure thanks for listening to this conversation with jitendra malik and thank you to our sponsors betterhelp and expressvpn please consider supporting this podcast by going to betterhelp.com lex and signing up at expressvpn.com lexpod click the links buy the stuff it's how they know i sent you and it really is the best way to support this podcast and the journey i'm on if you enjoy this thing subscribe on youtube review 5 stars on apple podcast support it on patreon or connect with me on twitter at lex friedman don't ask me how to spell that i don't remember myself and now let me leave you with some words from prince mishkin and the idiot by dostoyevsky beauty will save the world thank you for listening and hope to see you next time you