Jeremy Howard: fast.ai Deep Learning Courses and Research | Lex Fridman Podcast #35
the following is a conversation with Jeremy Howard he's the founder of fast AI a Research Institute dedicated to making deep learning more accessible he's also a distinguished research scientist at the University of San Francisco a former president of Kegel as well as the top ranking competitor there and in general he's a successful entrepreneur educator researcher and an inspiring personality in the AI community when someone asked me how do I get started with deep learning fast AI is one of the top places that point them to it's free it's easy to get started it's insightful and accessible and if I may say so it has very little BS they can sometimes dilute the value of educational content on popular topics like deep learning fast AI has a focus on practical application of deep learning and hands-on exploration of the cutting edge that is incredibly both accessible to beginners and useful to experts this is the artificial intelligence podcast if you enjoy it subscribe on YouTube give it five stars and iTunes supported on patreon or simply connect with me on Twitter Alex Friedman spelled Fri D ma N and now here's my conversation with Jeremy Howard what's the first program you've ever ridden this program I wrote that I remember would be at high school I did an assignment where I decided to try to find out if there were sand like better musical scales and the normal twelve tone twelve interval scale so I wrote a program on my Commodore 64 in basic let's search through other scale sizes to see if you could find one where they were more accurate you know harmonies like mid tone like sliding like he won an actual exactly 3 to 2 ratio where else with a 12 interval scale it's not exactly 3 to 2 for example so that's in the car well tempered as I say you know and basic on a Commodore 64 yeah where was the interest in music from or is it just I took music all my life so I played the phone and clarinet and piano and guitar and drums and whatever so how does that threat go through your life where's music today yeah it's not where I wish it was I for various reasons couldn't really keep it going particularly because I had a lot of problems with RSI with my fingers and so I had to kind of like cut back anything that used hands and fingers I hope one day I'll be able to get back to it health-wise so there's a love for music underlying it all yeah what's your favorite instrument sex the phone sex baritone saxophone well probably bass saxophone but they're awkward well I'm I always love it when music is coupled with programming there's something about a brain that utilizes those that emerges with creative ideas so you've used and studied quite a few programming languages can you given an overview of what you've used one of the pros and cons of each well my favorite programming environment almost certainly was Microsoft Access back in like the earliest days so that was Visual Basic for applications which is not a good programming language for the programming environment fantastic it's like the ability to create you know user interfaces and tie data and actions to them and create reports and all that as I've never seen anything as good there's things nowadays like air table which you're like small subsets of that which people love for good reason but unfortunately nobody's ever achieved anything like that what is that if you could pause in there for a second no access this is it a database database program that Microsoft produced part of office and the kind of wizard you know but basically it lets you in a totally graphical way create tables and relationships and queries and tie them to forms and set up you know event handlers and calculations and it was very plate powerful system designed for not massive scalable things but fair like useful little applications that I loved so what's the connection between excel and access so very close so access kind of was the relational database equivalent if you like so people still do a lot of that stuff it should be an access in Excel excels they don't know what Excel is great as well so but it's just not as rich a programming model as VBA combined with a relational database and so I've always loved relational databases but today programming on top of a relational database is just a lot more of a headache you know you generally either need to kind of you know you need something that connects that that runs some kind of database server unless you use circle light which has its own issues then you can often if you want to get a nice programming model you'll need to like create and add an ORM on top and then I don't know there's all these pieces tie together and it's just a lot more awkward than it should be there are people that are trying to make it easier so in particular I think of if sharp you know Don Syme who him and his team have done a great job of making something like a database appear in the type system so you actually get like tab completion for fields and tables and stuff like that anyway so that was kind of anyway so like that whole VBA office thing I guess was a starting point which I still miss I got into standard Visual Basic that's interesting just to pause on them for a second it's interesting that you're connecting programming languages to the ease of management of data yeah so in your use of programming languages you always had a love and a connection with data I've always been interested in doing useful things for myself and for others which generally means getting some data and doing something with it and putting it out there again so that's been my interest throughout so I also did a lot of stuff with Apple script back in the early days so it's kind of nice being able to get the computer and computers to talk to each other and to do things for you and then I could think that one night the programming language I most loved then would have been Delphi which was object pascal created by under sales berg who previously did to it by pascal and then went on to create dotnet and then went on create typescript delphi was amazing because it was like a compiled fast language that was as easy to use as Visual Basic Delphi what is it similar to in in more modern languages Visual Basic Visual Basic yeah that a compiled fast version so I'm not sure there's anything quite like it anymore if you took like C shop or Java and got rid of the virtual machine and replaced it with something you could compile a small type binary I feel like it's where um Swift could get to with the new Swift UI and the cross-platform development going on like that's one of my dreams is that will hopefully get back to where Delphi was there is actually a free Pascal project nowadays called Lazarus which is also attempting to kind of recreate Delphi though they're making good progress so ok Delphi that's one of your favorite programming languages programming environments again I hate Pascal's not a nice language if you wanted to know specifically about what languages I like they would definitely pick J there's being an amazingly wonderful language well woods j.j are you aware of APL I am NOT okay so from doing a little research on work you've done okay so not at all surprising you're not familiar with it cuz it's not well known but it's actually one of the main families of programming languages going back to the late 50s early 60s so there was a couple of major directions one was the kind of lambda calculus Alonzo Church direction which I guess kind of listens game and whatever which has a history going back to the early days of computing the second was the kind of imperative /o o you know algo Simula going under C C++ so forth there was a third which Accord array oriented languages which started with a paper by a guy called Ken Iverson which was actually a math theory paper not a programming paper it was called notation as a tool for thought and it was the development of a new way a new type of math notation and the idea is that this math notation would be was was much more flexible expressive and also well-defined then traditional math notation which is none of those things math notation is awful and so he actually turned that into a programming language and because this was the early 50s although that's very late 50s although names were available so he called his language a programming language or APL ABL APL is a implementation of notation as a tool for thought by which he means math notation and Ken and his son went on to do many things but eventually they actually produced you know a new language that was built on top of all the learnings of APL that was called J and J is the most expressive composable language of you know beautifully designed language I've ever seen this didn't have object-oriented components deserve that kind of thing there's not really it's an array oriented language it's a new it's a it's an it's it's the third half using array array oriented yes so I need to be a ray warrior so arranged it means that you generally don't use any loops but the whole thing is done with kind of a extreme version of broadcasting if you're familiar with that none got an umpire slash Python concept so you do a lot with one line of code it looks a lot like math notation basically I'll compact mm-hm and the idea is that you can kind of because you can do so much with one line of code a single screen of code is very unlikely to you very rarely need more than that to in the rest your program and so you can kind of keep it all in your head and you can kind of clearly communicate it it's interesting that the APL created two main branches k and j j is this kind of like open source niche community of crazy enthusiasts like me and then the other path k was fascinating it's an astonishingly expensive programming language which many of the world's most ludicrous a rich hedge funds use so the entire machine is so small it sits inside level 3 cache on your CPU and and it easily wins every benchmark I've ever seen in terms of data processing speed hey you don't come across it very much because it's like $100,000 per CPU to to run it yeah but it's like this this this this path of programming languages it's just so much that are not so much more powerful in every way than the ones that almost anybody uses every day so though it's all about computation it's really focused pretty heavily focused on computation I mean so much of programming is data processing by definition and so there's a lot of things you can do with it but yeah there's not much work being done on making like use user interface talking us or whatever I mean this some but it's they're not great at the same time you've done a lot of stuff with Perl and Python yeah so where does that fit into the picture of J and K and APO and well you know it's much more pragmatic like in the end you kind of have to end up where the where the libraries are you know like because to me my my focus is on productivity I just want to get stuff done and solve problems so Perl was great for I created an email company called fast mail and Perl was great cuz back in the late 90s early 2000s it just had a lot of stuff it could do I still had to write my own monitoring system and my own web framework my own whatever because like none of that stuff existed but it was the super flexible language to do that in and you used Perl fast ball used as a back-end think so everything was written in Perl yeah yeah everything everything was fell why do you think Perl hasn't succeeded or hasn't dominated the market where Python really takes over a lot yeah well I mean it felt did dominate it was for time everything everywhere but then the guy that Pal Larry will kind of just didn't put the time in anymore and no project can be successful if there isn't you know it's particularly one that's data with a strong leader that that loses that strong leadership so then python is kind of replaced - you know python is a lot less elegant language in nearly every way but it has the data science libraries and a lot of them are pretty great so I kind of use it because it's the best we have but it's definitely not good enough what do you think the future programming looks like what do you hope the future programming looks like if we zoom in on the computational fields on data science on machine learning I hope Swift is successful because the goal is Swift the way Chris Lattner describes it is to be infinitely hackable and that's what I want I want something where me and the people I do research with and my students can look at and change everything from top to bottom there's nothing mysterious and magical and inaccessible unfortunately with Python it's the opposite of that because pythons so slow it's extremely unhackable you get to a point where it's like okay from here on down at sea so your debugger doesn't works in the same way your profiler doesn't work in the same way your build system doesn't work in the same way it's really not very happy ball at all what's the part you would like to be hackable is it for the objective of optimizing training of neural networks inference in your networks is it performance of the system or is there some non performance related just it's it's a greater thing I'm in the end I want to be productive as a practitioner so that means that so like at the moment our understanding of deep learning is incredibly primitive there's very little we understand most things don't work very well even though it works better than anything else out there there's so many opportunities to make it so you look at any domain area like I don't know speech recognition with deep learning or natural language processing classification with deep learning or whatever every time I look at an area with deep learning I always see like oh it's terrible there's lots and lots of obviously stupid ways to do things that need to be fixed so then I want to be able to jump in there and quickly experiment and make them better using the programming language is has a role in a huge role yes so currently Python has a big gap in terms of our ability to innovate particularly around recurrent neural networks and natural language processing because it because it's so slow the the actual loop where we actually loop through words we have to do that whole thing in CUDA C so we actually can't innovate with the kernel the heart of that most important algorithm and it's just a huge problem and this happens all over the place so we hit you know research limitations another example convolutional neural networks which actually the most popular architecture for lots of things maybe most things in declining we almost certainly should be using space convolutional neural networks but only like two people are because to do it you have to rewrite all of that CUDA sea level stuff and yeah this researchers and practitioners don't so like there's just big gaps in like what people actually research on what people actually implement because of the programming language problem so you think you think it's it's just too difficult to write in CUDA see that a programming like a higher level programming language like Swift should enable the the easier input fooling around creative stuff with RN ends or was parse convolution your noise kind of who's a who's at fault who's who's a charge of making it easy for a research - player I mean no one's at fault just know what he's got around to it yet or it's just it's hard right and I mean part of the fault is that we ignored that whole APL kind of direction most prominently everybody did for 60 years 50 years but recently people have been starting to reinvent pieces of that and kind of create some interesting new directions in the compiler technology so the place where that's particularly happening right now is something called ml ir which is something that ok I'm Kris lat know this rift guy is leading and because it's actually not gonna be swift on its own that solves his problem because the problem is they're currently writing a acceptable fast you know GPU program is too complicated regardless of what language you use no and that's just because if you have to deal with the fact that I've got you know 10,000 threads and I have to synchronize between them all and I have to put my thing in to grid blocks and think about warps and all this stuff it's just it's just so much boilerplate to do that well you have to be a specialist at that and it's going to be a year's work to you know optimize that algorithm in that way but with things like tensor comprehensions and tile and ml ir and t vm there's all these various projects which are all about saying let's let people create like domain-specific languages for tensor computations these are the kinds of things we do are generally in on the GPU for deep learning and then have a compiler which can optimize that tensor computation a lot of this work is actually sitting on top of a project called halide which was is a mind-blowing project where they came up with such a domain-specific language in fact true one domain-specific language for expressing this is what my tensor computation is and another domain-specific language for expressing this is the kind of the way I want you to structure the compilation of that like do it block by block and do these bits in parallel they were able to show how you can compress the amount of code by 10x compared to optimized GPU code and get the same performance so that's like so these other things are kind of sitting on top of that kind of research and ml ir is pulling a lot of those best practices together and now we're starting to see work done on making all of that directly accessible through Swift so that I could use Swift to kind of write those domain-specific languages and hopefully we'll get them Swift CUDA kernels written in a very expressive and concise way that looks a bit like J in APL and then Swift layers on top of that and then a swift UI on top of that and you know it'll be so nice if we can get to that point that does it all eventually boil down to CUDA and NVIDIA GPUs unfortunately at the moment it does but one of the nice things about ml ir if AMD ever gets their act together which they probably won't is that they or others could write MLA our backends for other GPUs or other or other tensor computation devices of which today there are increasing number are like graph core or vertex AI or whatever so yeah being able to target lots of backends would be another benefit of this and the market really needs competitions at the moment NVIDIA is massively overcharging for their kind of enterprise class cards because there is no serious competition because nobody else is doing the software properly in the cloud there is some competition right but not really other than TP used for heavy use are almost unprogrammed well at the moment you can't the GPUs has the same problem the case is even worse so TP use the Google actually made an explicit decision to make them almost entirely unprogrammed ball because they felt that there was too much IP in there and if they gave people direct access to program them people would learn their secrets yeah so you can't actually directly program the memory in a teepee you you can't even directly like create code that runs on and that you look at on the machine that has the GPU it all goes through a virtual machine so all you can really do is this kind of cookie cutter thing of like plug into high-level stuff together which is just super tedious and annoying and totally unnecessary so what was the tell me if you could the origin story of fast AI what is the motivation its mission its dream so I guess the founding story is heavily tied in my previous startup which is a company called in lytic which was the first company to focus on deep learning for medicine and I created that because I saw that was a huge opportunity to there's a there's a about a 10x shortage of the number of doctors in the world and the developing world that we need expected it would take about three hundred years to train enough doctors to meet that gap but I guess that maybe if we used deep learning for some of the analytics we could maybe make it so you don't need as highly trained doctors diagnosis diagnosis and treatment planning where's the biggest benefit just before get the first day I was where's the biggest benefit of AI in medicine DC today and not much not much happening today in terms of like stuff that's actually out there it's very early but in terms of the opportunity it's to take markets like India and China and Indonesia which have big populations Africa small numbers of doctors and provide diagnostic particularly treatment planning and triage kind of on device so that if you do a you know test for malaria or tuberculosis or whatever you immediately get something that even a health care worker that's had a month of training can get a very high quality assessment of whether the patient might be at risk until you know okay we'll send them off to a hospital so for example in Africa outside of South Africa there's only five pediatric radiologists for the entire continent so most countries don't have any so if your kid is sick and they need something diagnose your medical imaging the person even if you're able to get medical imaging done the person that looks at it will be you know a nurse at best yeah but actually in India for example and in China almost no x-rays are read by anybody by any trained professional because they don't have enough so if instead we had a algorithm that could take the most likely high-risk 5% and say triage basically say okay somebody needs to look at this it would massively change the kind of way that what's possible with medicine in the developing world and remember they have increasingly they have money there the developing world they're not imported Apella people so they have the money so that they're building the hospitals they're getting the diagnostic equipment but they just there's no way for a very long time will they be able to have the expertise shortage of their sweeties okay and that's where the deep learning systems could step in and magnify the expertise they do exactly yeah so you do see just a longer it a little bit longer yeah the interaction you still see the human expert still at the core of these systems yeah absolutely there's something in medicine that can be automated almost completely I don't see the point of even thinking about that because we have such a shortage of people why would we not why would we want to find a way not to use them like we have people so the idea of like even from an economic point of view if you can make them 10x more productive getting rid of the person doesn't impact your unit economics at all and it totally ignores effect that there are things people do better than machines so it's just to me that's not a useful way of framing the problem I guess just to clarify I guess I meant there may be some problems where you can avoid even going to the expert ever sort of maybe preventive care or some basic stuff flowing and food allowing the expert to focus on the things that are that are really that well that's what the triage would do right so the triage would say okay it's ninety ninety nine percent sure there's nothing here right so you know that can be done on device and they can just say okay go home so the experts are being used to look at the stuff which has some chance it's worth looking at which most things is it's not you know it's fine why do you think we haven't quite made progress on that yet in terms of the the scale of how much AI is applied in the middle there's a lot of reasons I mean one is it's pretty new I only started and let it can like 2014 and before that like it's hard to express to what degree the medical world was not aware of the opportunities here so I went to iris na which is the world's largest radiology conference and I told everybody I could you know like I'm doing this thing this deep learning please come and check it out and no one had any idea what I was talking about and no one had any interest in it so like we've come from absolute zero which is hard and then the whole regulatory framework education system everything is just set up to think of doctoring in a very different way so today there is a small number of people who are deep learning practitioners and doctors at the same time and that we're starting to see the first ones come out of their PhD programs so that Kinane over in fostering Cambridge has a number of students now who are data data science experts deep learning experts and and actual medical doctors quite a few doctors have completed first day of course now and are publishing papers and creating journal reading groups in the American Council of radiology and like it's just starting out but it's going to be a long process they regulators have to learn how to regulate this they have to build you know guidelines and then the lawyers at hospitals have to develop a new way of understanding that sometimes it makes sense for data to be you know looked at in raw form in large quantities in order to create world-changing results he has a regulation around data all that it sounds it was probably the hardest problem but sounds reminiscent of autonomous vehicles as well many of the same regulatory challenges meaning the same data challenges yeah I mean funnily enough that problem is less their regulation and more the interpretation of that regulation by by lawyers in hospital so hipper is actually was designed to its it to P and hipper is not standing does not stand for privacy it stands for portability it's actually meant to be a way that data can be used and it was created with lots of gray areas because the idea is that would be more practical and would help people to use this this legislation to actually share data in a more thoughtful way unfortunately it's done the opposite because when a lawyer sees a gray area they see oh if we don't know we won't get sued then we can't do it today so hipper is not exactly the problem the problem is more than there's hospital lawyers are not incentive to make bold decisions about data portability or even to embrace technology that saves lives no they more want to not get in trouble for embracing the right but also it is also so slaves in a very abstract way which is like oh we've been able to release these hundred thousand and on most records I can't point at the specific person whose life that's saved I can say like oh we've ended up with this paper which found this result which you know diagnosed a thousand more people otherwise but it's like which ones were helped it's it's very abstract and on the counter side of that you may be able to point to a life that was taken because of something though yeah or or or a person whose privacy was violated it was like oh this specific person you know there was de-identified so we've identified just a fascinating topic we're jumping around I'll get back to fast AI but on the question of privacy data is the fuel for so much innovation in deep learning what's your sense and privacy whether we're talking about Twitter Facebook YouTube just the technologies like in the medical field that rely on people's data in order to create impact how do we get that right respecting people's privacy and yet creating technology that just learns from data one of my areas of focus is on doing more with less data which so most vendors unfortunately are strongly incented to find ways to require more data and more computation so Google and IBM being the most obvious IBM yeah so Watson you know so Google and IBM both strongly push the idea that you have to be you know that they have more data and more computation and more intelligent people than anybody else and so you have to trust them to do things because nobody else can do it and Google's very upfront about this like Geoff Dana's going out there and given talks and said our goal is to require a thousand times more computation but less people our goal is to use the people that you have better and the data you have better in the computation you have better so one of the things that we've discovered is or or at least highlighted is that you very very very often don't need much data at all and so the data you already have in your organization we'll be enough to get state-of-the-art results so like my starting point would be this going to say around privacy is a lot of people are looking for ways to share data and aggregate data but I think often that's unnecessary they assume that they need more data than they do because they're not familiar with the basics of transfer learning which is this critical technique for needing orders of magnitude less data is your sense one reason you might want to collect data from everyone is like in the recommender system context where your individual Jeremy Howard's individual data is the most useful for freeing for providing a product that's impactful for you so for giving you advertisements for recommending to your movies for doing medical diagnosis is your sense we can build with a small amount of data general models they will have a huge impact for most people that we don't need to have data from punching on the whole I'd say yes I mean they're things like you know recommender systems have this cold-start problem where you know Jeremy is a new customer we haven't seen him before so we can't recommend him things based on what else he's bought and liked with us and there's various workarounds to that like in a lot of music programs we'll start out by saying which of these artists you like which of these albums do you like which of these songs do you like Netflix used to do that nowadays they they tend not to people kind of don't like that because they think oh we don't want to bother the user so you could work around that by having some kind of data sharing where you get my marketing record from axiom or whatever and try to guess from that to me the the benefit to me and to society of saving me five minutes on answering some questions versus the negative externalities of if the privacy issue doesn't add up so I think like a lot of the time the places where people are invading our privacy in order to provide convenience is really about just trying to make them more money and and they move these negative externalities and to places that they don't have to pay for them so when you actually see regulations appear that actually cause the companies that create these negative externalities to have to pay for it themselves they say well we can't do it anymore so the cost is actually too high right but for something like medicine yeah I mean the hospital has my you know medical imaging my pathology studies my medical records and also I own my medical data so you can so I I helped a startup called doc AI one of the things doc AI does is that this has an app you can connect to you know Sutter Health's and webcore and Walgreens and download your medical data to your phone and then upload it again at your discretion to share it as you wish so with that kind of approach we can share our medical information with the people we want to yes of control I mean it really being able to control who you share with us on yeah so that that has a beautiful interesting tangent but to return back to uh the origin story of fast they act right so so before I started fast AI I spent a year researching where the biggest opportunities for deep learning because I knew from my time at Cal in particular that deep learning had kind of hit this threshold point where it was rapidly becoming the state of the art approach in every areas that looked at it and I've been working with neural nets for over 20 years I knew that from a theoretical point of view once it hit that point it would do that in kind of just about every domain and so I kind of spent a year researching what are the domains it's going to have the biggest low-hanging fruit in the shortest time period medicine but there were so many I could have picked and so there was a kind of level of frustration for me of like okay I'm really glad we've opened up the medical deep learning world and today is huge as you know but we can't do you know I can't do everything I don't even know like it took like in medicine it took me a really long time to even get a sense of like what kind of problems to medical practitioners solve what kind of data do they have who has that data so I kind of felt like I need to approach this differently if I want to maximize the positive impact of deep mourning rather than me picking an area and trying to become good at it and building something I should let people who are already domain experts in those areas and who already have the data do it themselves mm-hmm so that was the reason for fast AI is to basically try and figure out how to get deep learning into the hands of people who could benefit from it and help them to do so in as quick and easy and effective way as possible god it's all sort of empowered the the domain expert yeah and like partly it's because like unlike most people in this field my background is very applied and industrial that my first job at MIT was at McKinsey and company I spent 10 years in management consulting I I spend a lot of time with domain experts you know so I kind of respect them and appreciate them and know I know that's where the value generation in society is and so I also know how most of them can't code and most of them don't have the time to invest you know three years and a graduate degree or whatever so it's like how do i skill those two main experts I think it would be a super powerful thing you know biggest societal impact I could have so that yeah that was the thinking so so much a fast AI students and researchers and the things you teach are pragmatically minded right practically minded freaking figuring out ways how to solve real problems and fast right so from your experience what's the difference between theory and practice of deep learning well most of the research in the deep mining world is a total waste of time all right that's what I was getting at yeah it's it's a problem in science in general scientists need to be published which means they need to work on things that their peers are extremely familiar with and can recognize in advance in that area so that means that they all need to work on the same thing and so it really Inc and and the thing they work on there's nothing to encourage them to work on things that are practically useful so you get just a whole lot of research which is minor advances and stuff that's been very highly studied and has no significant practical impact where else the things that really make a difference like I mentioned transfer learning like if we can do better at transfer learning then it's this like world-changing thing we're suddenly like lots more people can do world-class work with less resources and less data and but almost nobody works on that or another example active learning which is the study of like how do we get more out of the human beings in the loop where's my favorite topic yeah so active learning is great but it's almost nobody working on it because it's just not a trendy thing right now you know what somebody's suicide interrupt you're saying that nobody is publishing an active learning but there's people inside companies anybody who actually has to solve a problem they're going to innovate an active learning yeah everybody kind of reinvents active learning when they actually have to work in practice because they start labeling things and they think gosh this is taking a long time and it's very expensive and then they start thinking well why am i labeling everything I'm only the machines only making mistakes on those two classes they're the hard ones maybe I ought to start labeling those two classes and then you start thinking well why did I do that manually why kind of just get the system to tell me which things are going to be hardest it's an obvious thing to do but yeah it's it's just like like transplant learning it's it's under studied and the academic world just has no reason to care about practical results the funny thing is like I've only really ever written one paper I hate writing papers and I didn't even write it it was my colleague sebastian ruder who actually wrote it I just knew did the research for it but it was basically introducing transfer learning successful transfer learning to NLP for the first time the algorithm is called GLM fit and it actually I actually wrote it for the course for the first day of course I wanted to teach people in LP and I thought I only want to teach people practical stuff and I think the only practical stuff is transfer learning and I couldn't find any examples of transfer learning and NLP so I just did it and I was shocked to find that as soon as I did it was you know the basic prototype took a couple of days smashed the state-of-the-art on one of the most important data sets in a field that I knew nothing about and I just thought well this is ridiculous and so I spoke to the best unit and he kindly offered to write it up the results and so it ended up being published in a CL which is the top link with a computational linguistics conference so like people do actually care once you do it but I guess it's difficult for maybe like junior researchers or like like I don't care whether I get citations or papers whatever I was right there's nothing in my life that makes that important which is why I've never actually bothered to write a pic of myself now for people who do I guess they have to pick the kind of safe option which is like yeah make a slight improvement on something that everybody is already working on yeah nobody does anything interesting or succeeds in life or the safe option speed I mean the nice thing is nowadays everybody is now working on you know a transfer learning because since that time we've had GPT and GPT too and Burt and you know it's like it's so yeah once you show that something is possible if nobody jumps you and I guess I hope to be a part of and I hope to see more innovation and active learning in the same way I think yeah try learning an active learning are fascinating public open were I actually helped start a startup called platform AI which is really all about active learning and yeah it's very interesting trying to kind of see what research is out there and make the most of it and there's basically none so we've had to do all our own research once again and just as easy described can you tell the story of the stanford competition dawn bench and fast day eyes achievement on it sure so something which I really enjoy is that I basically teach two courses a year the practical deep money for coders which is kind of the introductory course and then cutting-edge tech mining for coders which is the kind of research level course and while I teach those courses I have a I basically have a big office at the University of San Francisco big enough for like 30 people and I invite anybody any student who wants to come and hang out with me well I built the course and so generally it's full and so we have twenty or thirty people in a big office with nothing to do but study deep learning so it was during one of these times that somebody in the group said oh there's a thing called Don benched it looks interesting and I was like what the hell is that is it about some competition to see how quickly you can train a model seems kind of not exactly relevant to what we're doing but it sounds like the kind of thing which you might be interested in I checked it out and I said oh crap there's only ten days till it's over it's pretty too late and we're kind of busy trying to teach this course yeah maybe like oh it would make an interesting case study for the course like it's all the stuff where you're already doing why don't you just put together our current best practices and ideas so me and I guess about four students just decided to give it a go and we focused on this more one called Sipho ten which is that all 32 by 32 pixels can you say word on benches yeah so it's a competition to train a model as fast as possible I was run by Stanford as cheap as possible - that's also another one first cheap as possible and there was a couple of categories imagenet and so far 10 so image nets is big 1.3 million image thing that took a couple of days to train remember a friend of mine Pete worden who's now at Google I remember he told me how he trained imagenet a few years ago and he basically like had this little granny flat out the back that he turned into his image net training center and he figured you know after like a year of work he figured out how to train it and like ten days or something it's like that was a big job well so far ten at that time you could train in a few hours you know it's much smaller and easier so we thought would try so far 10 and yeah I've really never done that before like I've never really liked things like using more than one gpgpu at a time was something I tried to avoid cuz to me it's like very against the whole idea of accessibility is she better to do things with 1gb here I mean have you asked in the past before after having accomplished something how do I do this faster much faster Oh always but it's always for me it's always how do I make it much faster on a single genus you that a normal person could afford in their day-to-day life it's not how could I do it faster I you know having a huge data center because up to me it's all about like as many people should be to use something as possible without fussing around with infrastructure so anyway so in this case it's like well we can use eight GPUs just by renting a AWS machine so we thought we'd try that and yeah basically using the stuff we were already doing we were able to get you know the speed you know within a few days we had to speed down to I don't know that's a very small number of minutes I can't remember exactly how many minutes it was but I might have in like 10 minutes or something and so yeah we found ourselves at the top of the leaderboard easily for both time and money which really shocked me because the other people competing this were like Google and Intel and stuff we're like know a lot more about this stuff I think we do so that we were emboldened we thought let's try the imagenet one two way out of our league but our goal was to get under 12 hours yeah and we did which was really exciting and but we didn't put anything up on the leaderboard but we were down to like 10 hours but then Google put in some like 5 hours or something about us like oh they're so screwed but we kind of thought we'll keep trying you know if Google can do it info I mean Google did on five hours on someone like a TPU pod or something like a lot of hardware but we kind of like had a bunch of ideas to try like a really simple thing was why are we using these big images they're like 224 256 by 256 pixels you know why don't we try smaller ones and just elaborate there's a constraint on the accuracy that your training model is supposed to achieve yeah you got to achieve 93% I think it was for imagenet exactly which is very tough so you have to yeah 93% like they think that they picked a good threshold it was a little bit higher than what the most commonly used ResNet 50 model could achieve at that time so yeah so it's quite a difficult problem to solve but yeah we realized if we actually just use 64 by 64 images it trained a pretty good model and then we could take that same model and just give it a couple of epochs to learn 224 by 224 images and it was basically already trained it makes a lot of sense like if you teach somebody like here's what a dog looks like and you show them low res versions and then you say here's a really clear picture of a dog they already know what a dog looks like so that like just we jumped to the front and we ended up winning parts of that competition we actually ended up doing a distributed version over multiple machines a couple of months later and ended up at the top of the leaderboard we had 18 minutes in it yeah and it was and people have just kept on blasting through again and again since then so so what's your view on multi-gpu or multiple machine training in general as as a way to speed code up I think it's largely a waste of time both multi-gpu on a single machine and yeah particularly multi machines because it's just clunky motogp use is less clunky than it used to be but to me anything that slows down your iteration speed is a waste of time so you could maybe do your very last you know perfecting of the model on Motty GPUs if you need to that so for example I think doing stuff on imagenet is generally a waste of time why test things on 1.3 million images most of us don't use 1.3 million images and we've also done research that shows that doing things on a smaller subset of images gives you the same relative answers anyway so from a research point of view why waste that time so actually I released a couple of new data sets recently one is called imaginet the French image net which is a small subset of image net which is designed to be easy to classify I would highly spell imaginer it's got an extra T and e at the end because it's very French am i okay yeah I'm okay and then another one called image Wharf which is a subset of the image net that only contains dog breeds that's a hard one right that's a hard one yeah and I've discovered that if you just look at these two subsets you can train things on a single GPU in ten minutes and the results you get directly transferable to imagenet nearly all the time and so now I'm starting to see some researchers start to use these holidays that's so deeply love the way you think because I think you might have written a blog post saying that sort of going these big data sets is encouraging people to not think creatively absolutely so you're - it's sort of constrained you to Train on large resources and because you have these resources you think more research will be bit better and then you start like for some somehow you kill the creativity yeah and even worse than that Lex I keep hearing from people who say I decided not to get into deep learning because I don't believe it's accessible to people outside of Google to do useful work so like I see a lot of people make an explicit decision to not learn this incredibly valuable tool because they've they've drunk the Google kool-aid which is that only Google's big enough and smart enough to do it and I just find that so disappointing and it's so wrong and I think all the major breakthroughs in AI in the next twenty years will be doable on a single GPU like I would say my sense is all the big sort of well let's put it this way none of the big breakthroughs of the last 20 years or acquired multiple GPUs so like fetch norm well you drop out did you demonstrate to everyone of them yeah this is five multiple GPUs against the original Gans didn't require multiple ups well and and we've actually recently shown that you don't even need gains so we've developed gained level outcomes without knitting Gans and we can now do it with again by using transfer learning we can do it in a couple of hours on a single generator might like without the other serial port yeah so we've found loss functions that work super well without the adversarial part and then one of our students guy called Jason antic has created Cordiale defi which uses this technique to colorize old black-and-white movies you can do it on a single GPU color as a whole movie in a couple of hours and one of the things that Jason and I did together was we figured out how to add a little bit of n at the very end which it turns out for colorization makes it just a bit brighter and nicer and then Jason did masses of experiments to figure out exactly how much to do but it's still all done on his home machine on a single GPU in his lounge room and like if you think about like colorizing Hollywood movies that sounds like something a huge studio it would have to do but he has the world's best results on this there's this problem of microphones we're just talking two microphones now yeah it's such a pain in the ass to have these microphones to get good quality audio and I tried to see if it's possible to plop down a bunch of cheap sensors and reconstruct higher quality audio from multiple sources because right now I haven't seen work from okay we can say inexpensive mics automatically combining audio from multiple sources to improve the combined audio right people haven't done that and that feels like a learning problem alright so hopefully somebody can well I mean it's it's eminently doable and it should have been done by now I feel I felt the same way about computational photography four years ago that's right why are we investing in big lenses when three cheap lenses plus actually a little bit of intentional movement so like Holden you don't like take a few frames gives you enough information to get excellent sub pixel resolution which particularly with deep learning you would know exactly what you meant to be looking at we can totally do the same thing with audio I think there's a madness that it hasn't been done yet I live in progress on the photographer tog Rafik um yeah the dog photography is basically standard now so the the Google picks all night light I don't know if you've ever tried it but it's it's astonishing you take a picture in almost pitch black and you get back a very high quality image and it's not because of the lens same stuff is like adding the bouquet to the you know the background wearing have done computationally this depicts over here yeah basically the everybody now is doing most of the fanciest stuff on their phones with computational photography and also increasingly people are putting more than one lens on the back of the camera so the same will happen for audio for sure and there's applications in the audio side if you look at an Alexa type device most people have seen especially I worked at Google before when you look at noise background removal you don't think of multiple sources of audio you don't play with that as much as I would hope people I mean you can still do it even with one like again it's not not much works being done in this area so we're actually going to be releasing an audio library soon which hopefully will encourage development of this because it's so underused the basic approach we used for our super resolution in which Jason uses video defy of generating high quality images the exact same approach would work for audio no-one's done it yet but it would be a couple of months work okay are also learning rate in terms of Don bench there's some magic on learning rate that you played around with yeah interesting yeah so this is all work that came from a guy called Leslie Smith Leslie's a researcher who like us cares a lot about just the practicalities of training neural networks quickly and accurately which i think is what everybody should care about but almost nobody does and he discovered something very interesting which he calls super convergence which is there are certain networks that with certain settings of high parameters could suddenly be trained ten times faster by using a ten times higher learning rate now no one published that paper because it's not an area of kind of active research in the academic world no academics recognized this is important and also deep learning in academia is not considered a experimental science so unlike in physics where you could say like I just saw as a subatomic particle do something which the theory doesn't explain you could publish that without an explanation and then in the next 60 years people can try to work out how to explain it we don't allow this in the deep learning world so it's it's literally impossible for Leslie to publish a paper that says I've just seen something amazing happen this thing trained ten times faster than it should have I don't know why and so the reviewers were like we can't publish that because you don't know why so anyway that's important to pause on because there's so many discoveries that would need to start like that every every other scientific field I know of works is that way I don't know why ours is uniquely disinterested in publishing unexplained experimental results but there it is so it wasn't published having said that I read a lot more unpublished papers and published papers because that's where you find the interesting insights so I absolutely read this paper and I was just like this is astonishingly mind-blowing and weird and awesome and like why isn't everybody only talking about this because like if you can train these things ten times faster they also generalized better because you're you're doing less epochs which means you look at the data less you get better accuracy so I've been kind of studying that ever since and eventually Leslie kind of figured out a lot of how to get it's done and we added minor tweaks and a big part of the trick is starting at a very low learning rate very gradually increasing it so as you're training your model you would take very small steps at the start and it gradually makes them bigger and bigger and tall eventually you're taking much bigger steps than anybody thought as possible a few other little tricks to make it work but ever ever basically we can reliably get super convergence and so for the dawn bench thing we were using just much higher learning rates than people expected to work what do you think the future of I mean makes so much sense for that to be a critical hyper parameter learning rate that you very what do you think the future of learning rate magic looks like well there's been a lot of great work in the last 12 months in this area it's and people are increasingly realizing that up to might like we just have no idea really how optimizers work and the combination of weight decay which is how we regularize optimizers and the learning rate and then other things like the epsilon we use in in the atom optimizer they all work together in weird ways and different parts of the model this is another thing we've done a lot of work on is research into how different parts of the model should be trained at different rates in different ways so we do something we call discriminative learning rates which is really important particularly for transfer learning so really I think in the last 12 months a lot of people have realized that this all this stuff is important there's been a lot of great work coming out and we're starting to see algorithms here which have very very few dials if any that you have to touch selector I think what's going to happen is the idea of a learning rate well it almost already has disappeared in the latest research and instead it's just like you know we we know enough about how to interpret the gradients and the change of gradients we see to know how to set every parameter you can await it so you see the future of of deep learning where really where's the input of a human expert needed well hopefully the input of the human expert will be almost entirely unneeded from the deep learning point of view so again like Google's approach to this is to try and use thousands of times more compute to run lots and lots of models at the same time and hopefully one of them is good at or male CONUS yeah I don't know kind of stuff which i think is insane when you better understand the mechanics of how models learn you don't have to try thousand different models to find which one happens to work the best you can just jump straight to the best one which means that it's more accessible in terms of compute cheaper and also with less hyper parameters to set it means you don't need deep learning experts to train your deep learning model for you which means that domain experts can do more of the work which means that now you can focus the human time on the kind of interpretation data gathering identifying what all errors and stuff like that yeah the data side how often do you work with data these days in terms of the cleaning looking at like Darwin looked at different species while traveling about do you look at data I have you in your roots and cargo always yeah good data I mean it's a key part of our course it's like before we train a model in the course we see how to look at the data and then after the first thing we do after we train our first model which we fine-tune an image net model for five minutes and then the thing we immediately do after that is we learn how to analyze the results of the model by looking at examples of misclassified images and looking at a classification matrix and then doing like research on Google to learn about the kinds of things that it's misclassifying so to me one of the three cool things about machine learning models in general is that you can interpret when you interpret them they tell you about things like what are the most important features which groups you misclassifying and they help you become a domain expert more quickly because you can focus your time on the bits that the model is telling you it is important so it lets you deal with things like data leakage for example if it says all the main feature I'm looking at is customer ID you know and you're like oh customer ID should be predictive and then you can talk to the people that manage customer IDs and they'll tell you like oh yes as soon as a customer's application is accepted we add a one on the end of their customer arm or something you know yeah so yeah model looking at data particularly from the lens of which parts of the date of the model says is important is super important yeah and using kind of using the model to almost debug the data yeah you have learn more about exactly what are the different cloud options for training y'all networks it's the last question related to dawn bench well it's part of a lot of the work we do but from a perspective of performance I think you've written this in a blog post there's AWS there's TPU from Google what's your sense what the future holds what would you recommend now right there was a so from a halfway point of view Google's TP use and the best nvidia gpus are similar I mean maybe the TP is like 30% faster but they're also much harder to program with there isn't a clear leader in terms of hardware right now although much more importantly the GPU nvidia gpus a much more programmable they've got much more written for all them so like that's the clear leader for me and where I would spend my time as a researcher and practitioner millington to the platform I mean we're super lucky now with stuff like Google TCP Google Cloud and AWS that you can access a GPU pretty quickly and easily but I mean for AWS it's still too hard like you have to find an ami and get the instance running and then install the software you want blah blah blah GCP is still is currently the the best way to get started on if the server environment because they have a fantastic fast AI in pi torch ready to go instance which has all the courses pre-installed it has Jupiter notebook pre running Jupiter notebook is this wonderful interactive computing system which everybody basically should be using for any kind of data-driven research but then even better than there are there are platforms like salamander which we own and paper space where literally you click a single button and it pops up a Jupiter notebook straight away without any kind of installation or anything and all the course notebooks are all pre-installed so like for me we this is one of the things we spent a lot of time kind of curating and working on because when we first started our courses the biggest problem was people dropped out of lesson one because they couldn't get an AWS instance running so things are so much better now and like we actually have if you got a cost up faster day I the first thing it says is here's how to get started with your GPU and there's like you just click on a link and you click start and and it's going it will you a go GCP I have to confess I've never used the Google DCP yeah JCP gives you three hundred dollars of compute for free which is really nice that as I say a salamander and paper spacer even even easier still okay so the from the perspective of deep learning frameworks you work with fast AI to go to this framework and PI torch intensive flow what are the strengths of each platform your perspective so in terms of what we've done our research on and taught in our course we started with Theano and care us and then we switch to tensor flow and care us and then we switch to PI torch and then we switched to PI torch and fast AI and that that kind of reflects a growth and development of the ecosystem of dig learning libraries siano intensive flow were great but we're much harder to teach and do research and development on because they define what's called a computational graph upfront less data graph well you basically have to say here are all the things that I'm going to eventually do in my model and then later on you say okay do those things with this data and you can't like debug them you can't do them step-by-step you can't program them interactively in a Jupiter notebook and so forth pi torch was not the first four pi torch was certainly the the strongest entrant to come along and say let's not do it that way let's just use normal Python and everything you know about in Python is just going to work and we'll figure out how to make that run on the GPU as in when and necessary that turned out to be a huge a huge leap in terms of what we could do with our research and what we could weigh with our teaching and because it was a limiting yeah I mean it was critical for us for something like dawn Bench to be able to rapidly try things it's just so much harder to be a researcher and practitioner when you have to do everything up front and you can inspect it problem with pay torch is it's not at all accessible to newcomers because you have to like write your own training loop and manage the gradients and all their stuff and it's also like not great for researchers because you're spending your time dealing with all this boilerplate and overhead rather than thinking about your algorithm so we ended up writing this very multi-layered API that at the top level you can train a state-of-the-art neural network in three lines of code and which kind of talks to an API which talks to an API which talks from API which like you can deep dive into at any level and get progressively closer to the Machine kind of levels of control and this is the first AI library that's been critical for us and for our students and for lots of people that have one big learning competitions with it and written academic papers with it it's made a big difference we're still limited though by Python and particularly this problem with things like recurrent neural nets a where you just can't change things unless you accept it going so slowly that it's impractical so in the latest incarnation of the course and with some of the research risked out now starting to do we're starting to do stuff some stuff in Swift I think we're three years away from that being super practical but I'm in no hurry I'm very happy to invest the time to get there but you know with with that we actually already have a nascent version of the first AI library for vision running on special knowledge and so flow because a Python for tensorflow is not going to cut it it's just a disaster what they did was they tried to replicate the bits that people were saying they like about a torch the is kind of interactive computation but they didn't actually change their foundational runtime components so they kind of added this like syntax sugar they call TF eager tend to flow again which makes it look a lot like pay torch but it's 10 times slower than pi torch to actually hmm do a step so because they didn't invest the time and like retooling the foundations cuz their code base is so horribly copy yeah I think it's probably very difficult to do that kind of rejoin yeah well particularly the way tensorflow was written it was written by a lot of people very quickly in a very disorganized way so like when you actually look in the code as I do it often I'm always just like oh god what were they thinking it's just it's pretty awful so I'm really extremely negative about the potential future if it by the flaws of the fet swift for tensorflow can be a different beast altogether it can be like it can basically be a layer on top of M lar that takes advantage of you know all the great compiler stuff that Swift builds on with LLVM and yeah it could be a thing kit will be absolutely fantastic well you're inspiring me to try evan Roo truly felt the pain of tensorflow 2.0 python it's fine by me but yeah but it does the job if you're using like predefined things that somebody's already written but if you actually compare you know like I've had to do because I've been having to do a lot of stuff with tensorflow recently you actually compare like okay I want to write something from scratch yeah like I just kick fighting is like oh it's running ten times slower than pi torch so is the biggest cost let's throw running time out the window how long it takes you to program that's not too different now thanks to transfer flow eager that's not too different but because because so many things take so long to run yeah you wouldn't run it at ten times slower like you just go like oh this is taking so long yeah and also there's a lot of things which are just less programmable like TF data which is the way they do processing works intensive flow is just this big mess it's incredibly inefficient and they kind of had to write it that way because of the TPU problems I described earlier so I just you know I just feel like they've got this huge technical debt which they're not gonna solve without starting from scratch so here's an interesting question then if there's a new student starting today what would you recommend they use well I mean we obviously recommend fast AI and pi torch because we teach new students and that's what we teach with so we would very strongly recommend that because it will let you get on top of the concepts much more quickly so then you'll become an extra and you'll also learn the actual state-of-the-art techniques you know so you actually get world-class results honestly it doesn't much matter what library you learn because switching from China to MX net to tensorflow to PI torch is going to be a couple of days work as few long as you understand the foundation as well but you think we'll Swift creep in there as a thing that people start using not for a few years particularly because like Swift has no data science community libraries Oh basil wing and the Swift community has a a total lack of appreciation and understanding of numeric computing so like they keep on making stupid decisions you know for years they've just done dumb things around performance and prioritization that's clearly changing now because the developer of Chris Christie at developer of Swift Chris Latner is working at Google on the Swift Potenza flows so like that's that's a priority it'll be interesting to see what happens with Apple because like Apple hasn't shown any sign of caring about numeric programming in Swift so I mean hopefully they'll get off their ass and start appreciating this because currently all of their low-level libraries are not written in Swift they're not particularly swifty at all stuff like whore ml they're really pretty rubbish so yeah so there's a long way to go but at least one nice thing is that Swift for tensorflow can actually directly use Python code and Python libraries you know literally the entire lesson one notebook a fast AI runs in Swift right now in Python mode so that's that's a nice intermediate thing how long does it take the look at the two two facile courses how long does it take to get from point zero to completing both courses it varies a lot somewhere between two months and two years generally so for two months how many hours a day so I sound like a somebody who is a very competent coder can can do 70 hours per course and seventy seven zero yeah that's it okay but a lot of people I know take a year off to study first day I full-time and say at the end of the year they feel pretty competent because generally there's a lot of other things you do like they're generally they'll be entering cowgirl competitions they you might be reading in Goodfellows books they might you know they'll be doing a bunch of stuff and often you know particularly if they are domain expert they're coding skills might be a little on the pedestrian side so part of it's just like doing a lot more writing what do you find is the bottleneck for people usually except getting started and setting stuff up I would say coding just yeah I would say the best the people who are strong coders pick it up the best although another bottleneck is people who have a lot of experience of classic statistics can really struggle because it the intuition is so the opposite of what they used to they're very used to like trying to reduce the number of parameters in their model and looking at individual coefficients and stuff like that so I find people who have a lot of coding background and know nothing about statistics are generally going to be the best off so you taught several course on deep learning and as Fineman says the best way to understand something is to teach it what have you learned about deep learning from teaching it a lot it's a key reason for me to to teach the courses I mean obviously it's going to be necessary to achieve our goal of getting two main experts to be familiar with deep learning but it was also necessary for me to achieve my goal of being really familiar with deep learning I I mean to see so many domain experts from so many different backgrounds it's definitely I wouldn't say taught me but convinced me something that I like to believe was true which was anyone can do it so there's a lot of kind of snobbishness out there about only certain people can learn to code only certain people are going to be smart enough to like do AI that's definitely bullshit you know I've seen so many people from so many different backgrounds get state-of-the-art results in their domain areas now the it's definitely taught me that the key differentiator between people that succeed and people that fail is tenacity that seems to be basically the only thing that matters the people a lot of people give up and but if the ones who don't give up pretty much everybody succeeds you know even if at first I'm just kind of like thinking like wow they're really not quite getting it yet are they but eventually people get it and they succeed so I think that's been any they're both things I'd like to believe was true but I don't feel like I really had strong evidence with them to be true but now I can say I've seen it again and again so what advice do you have for someone who wants to get started in deep learning train lots of models that's that's how you that's how you learn it so like so I would you know I think it's not just me I think I think our course is very good but also lots of people independently I said it's very good it recently won the cog X award for AI courses as being the best in the world let's say come to our course cost up faster day I and the thing I keep on hopping on in my lessons is train models print out the inputs to the models print out to the outputs to the models like study you know change change the inputs of it look at how the outputs very just run lots of experiments to get a you know an intuitive understanding of what's going on to get hooked do think you mentioned training do you think just running the models inference like if we talk about getting started no you've got to find cheering the models so that's that's that's the critical thing because at that point you now have model that's in your domain area so there's there's there's no point running somebody else's model because it's not your model like so it only takes five minutes to fine-tune a model for the data you care about and in lesson two of the course we teach you how to create your own data set from scratch by scripting Google Image Search yeah so and we show you how to actually create a web application running online so I create one in the course that differentiates between a teddy bear or grizzly bear and a brown bear and it does it with basically a hundred percent accuracy took me about four minutes to scrape the images from Google search in the script there's a little graphical widgets we have in the notebook that help you clean up the data set there's other widgets that help you study the results to see where the errors are happening and so now we've had got over a thousand replies in our share your work here thread of students saying here's the thing I built and so those people who like and a lot of them are state of the art like somebody said oh I tried looking at Devon Gehry characters and I couldn't believe it the thing that came out was more accurate than the best academic paper after lesson one and then there's others which are just more kind of fun like somebody who's doing Trinidad and Tobago hummingbirds she said that's kind of their national bird and she's got something that can now classify Trinidad and Tobago hummingbirds so yeah train models fine-tune models with your data set and then study their inputs and outputs how much is fast there of course is free everything we do is free we have no revenue sources of any kind it's just a service to the community you're a saint okay once the person understands the basics trains a bunch of models if we look at the scale of years what advice do you have for someone wanting to eventually become an expert train lots of models train lots of models in your domain area so an expert what right we don't need more expert like create slightly evolutionary research an area that everybody's studying we need experts at using deep learning to diagnose malaria well we need experts at using deep learning to analyze language to study media bias so we need experts in analyzing fisheries to identify problem areas and you know the ocean you know that that's that's what we need so like become the expert in your passion area and this is a tool which you can use just about anything and you'll be able to do that thing better than other people particularly by combining it with your passion and domain expertise so that's really interesting even if you do want to innovate on transfer learning or active learning your thought is that means one I certainly share is you also need to find a domain or dataset that you actually really care for right if you're not working on a real problem that you understand how do you know if you're doing it any good you know how do you know if your results so good how do you know if you're getting bad results why you're getting bad results is it a problem with the data or is like how do you know you're doing anything useful yeah the only to me the only really interesting research is not the only but the vast majority of interesting research is like try and solve an actual problem and solve it really well so both understanding sufficient tools and the deep learning side and becoming a domain expert in a particular domain I really thinks will then reach for anybody yeah I mean to me I would compare it to like studying self-driving cars having never looked at a car or being in a car or turn the car on right you know which is like the way it is for a lot of people they'll study some academic data set where they literally have no idea about the other way I'm not sure how familiar with the thomas vehicles but that is literally you describe a large percentage of robotics folks working in a self-driving cars as they actually haven't considered driving they haven't actually looked at what driving looks right they haven't driven it goes and enterprise because you know when you've actually driven you know like these are the things that happened to me and I was driving it so there's nothing that beats the real-world examples are just experiencing them you've created many successful startups what does it take to create a successful startup same thing is becoming successful deep learning practitioner which is not getting up so you can right out of money or time or run out of something you know but if you keep costs super low and try and save up some money beforehand so you can afford to have some time then just sticking with it it's one important thing doing something you understand and care about is important that by something I don't mean the biggest problem I see with deep whining people is they do a PhD in deep learning and then they try and commercialize their PhD it is a waste of time because that doesn't solve an actual problem you picked your PhD topic because it was an interesting kind of engineering or math or research exercise but yeah if you've actually spent time as a recruiter and you know that most of your time was spent sifting through resumes and you know that most of the time you're just looking for certain kinds of things and you can try doing that with a model for a few minutes and see whether that something which your models be able to do as well as you could then you're on the right track to creating a startup and then I think just yeah being just be pragmatic and train state to capital money as long as possible preferably forever so yeah on that point do you venture capital so did you were able to successfully run startups with was self-funded yeah my first two was self-funded and that was the right way to do it that's scary no species startups are much more scary because you have these people on your back who do this all the time and who have done it for years telling you grow grow grow grow and I don't they don't care if you fail they only care if you don't grow fast enough so that's scary where else doing the ones myself well with with partners who were friends is nice because like we just went along at a pace that made sense and we were able to build it to something which was big enough that we never had to work again but was not big enough that any VC would think it was impressive and that was enough for us to be excited you know so I I thought that's a much better way to do things and most people in generally speaking that for yourself but how do you make money during that process do you cut into savings if I guess so yeah so fir so I started fast mail and optimal decisions at the same time in 1999 with two different friends and for fast mail I guess I spent $70 a month on the server and when the server ran out of space I put a payments button on the front page and said if you want more than 10 makerspace you have to pay $10 a yeah and so run low like keep your cost down yes I came across town and once you know once once I needed to spend more money I asked people to spend the money for me and that that was that basically from then on oh we were making money and I was profitable from then for optimal decisions it was a bit harder because we were trying to sell something that was more like a 1 million dollar sale but what we did was we would sell scoping projects so kind of like prototype he projects but rather than to be free we would sell them 50 to $100,000 so again we were covering our costs and also making the client feel like we were doing something valuable so in both cases we were profitable from six months in yeah nevertheless is scary I mean yeah sure it's it's Gary before you jump in and I just I guess I was comparing it to this scariness of VC I felt like with VC stuff it was more scary you kind of much more in somebody else's hands you know will they fund you or not and what do they think of what you're doing I also found it very difficult with VC's bet startups to actually do the thing which I thought was important for the company rather than doing the thing which I thought would make the VC happy now VCS always tell you not to do the thing that makes them happy but then if you don't do the thing that makes them happy they get set so and do you think optimizing for the whatever they call it they exit is uh as a good thing to optimize for I think I can be but not at the VC level because the VC exit needs to be you know a thousand x so where else the lifestyle exit if you can sell something for ten million dollars I think you've made it right so I don't it depends if you want to build something that's gonna you kind of happy to do forever then fine if you want to build something you want to sell then three is time that's fine too I mean they're both perfectly good outcomes so you're learning Swift now in a way I mean you were a writer and I read that you use at least in some cases spaced repetition as a mechanism for learning new things yeah I used Anki quite a lot yourself sure I actually don't never talk to anybody about it don't don't know how many people do it but it works incredibly well for me can you talk to your experience like how did you what what do you like first of all okay let's back it up what is space repetition so spaced repetition is an idea created by a psychologist named Epping house must be a couple hundred years ago or something hundred and fifty years ago he did something which sounds pretty damn tedious he wrote down random sequences of letters on cards and tested how well he would remember those random sequences a day later or a week later whatever he discovered that there was this kind of a curve where his probability of remembering one of them would be dramatically smaller the next day and then a little bit smaller the next day a little bit smaller next day what he discovered is that if he revised those cards after a day the probabilities would decrease at a smaller rate and then if he revised them again a week later they would decrease it a smaller rate again and so he basically figured out a roughly optimal equation for when you should revise something you want to remember so spaced repetition learning is using this simple algorithm just something like revise something after a day and then three days and then a week and then three weeks and so forth and so if you use a program like Anki as you know it will just do that for you and if you and it will say did you remember this and if you say no it will reschedule it back to be up here again like ten times faster than it otherwise would have it's a kind of a way of being guaranteed to learn something because by definition if you're not learning it it will be rescheduled to be revised more quickly unfortunately though it's also like it doesn't let you for yourself if you not learning something you you know like it your revisions will just get more and more so you have to find ways to learn things productively and effectively like treat your brain well so using like mnemonics and stories and context and stuff like that so yeah it's it's a super great technique is like learning how to loan is something which everybody should learn before they actually learn anything but almost nobody does what have you so certainly works well for learning new languages for I mean for learned like small projects almost but do you you know I started using it for if you had who wrote a blog post about this inspired me I went Ben you I'm not sure is I started when I read papers all all concepts and ideas I'll put them was it Michael Nelson in my Illinois muscle strains that Michael started doing this recently and he's been writing about it I so the kind of today's evening house is a guy called Peter was niak who developed a system called super memo and he's been basically trying to become like the world's greatest Renaissance man over the last few decades he's basically lived his life with spaced repeated repetition learning for everything I and sort of like Michaels only very recently got into this but he started really getting excited about doing it for a lot of different things for me personally I actually don't use it for anything except Chinese and the reason for that is that Chinese is specifically a thing I made a conscious decision that I want to continue to remember even if I don't get much of a chance to exercise it because like I'm not often in China so I I don't or else something like programming languages or papers I have a very different approach which is I try not to learn anything from them but instead I try to identify the important concepts and like actually ingest them so like really understand that concept deeply and study it carefully I will decide if it really is important if it is like incorporated into our library you know incorporated into how I do things or decide it's not worth it say so I find I find I didn't remember the things that I care about because I'm using it all the time so I've fell at last 25 years I've committed to spending at least half of everyday learning or practicing something new which is all my colleagues have always hated because it always looks like I'm not working I mean if what I meant to be working on but it always means I do everything faster because I've been practicing a lot of stuff so I kind of give myself a lot of opportunity to practice new things and so I find now I don't yeah I don't often kind of find myself wishing I could remember something because if it's something that's useful then I've been using it a lot that's easy enough to look it up on google fit speaking Chinese you can't look it up on Google so do you have advice for people learning new things so if you what have you learned is a process does it I mean it all starts is just making the hours in the day available yeah you gotta stick with it which is again the number one thing that 99% of people don't do so the people I started learning Chinese with none of them were still doing it twelve months later I'm still doing a ten years later I tried to stay in touch with them but they just no one did it yeah for something like Chinese like study how human learning works so my every one of my Chinese flashcards is associated with a story and that story is specifically designed to be memorable and we find things memorable which are like funny or disgusting or sexy or related to people that we know will care about so I try to make sure all those stories that are in my head have those characteristics yeah so you have to you know you won't remember things well if they don't have some context and yeah you won't remember them well if you don't regularly practice them whether it be just part of your day to day life or the Chinese for me flashcards I mean the other thing is I'll let yourself fail sometimes so like I've had various medical problems over the last few years and basically my flashcards just stopped for about three years and then they've been other times I've stopped for a few months and it's so hard because you get back to it and it's like you have 18,000 cards June and so you just have to go alright well I can either stop and give up everything or just decide to do this every day for the next two years until I get back to it the amazing thing has been that even after three years I you know the Chinese were still in there like yeah it was so much faster to relearn than it was to learn the first time yeah absolutely it's it's in there the same with with guitar with music and so on it's sad because the work sometimes takes away and then you won't play for a year but really if you then just get back to it every day you're right through right there again what do you think is the next big breakthrough in artificial intelligence what are your hopes in deep learning or beyond that people should be working on or you hope there'll be breakthroughs I don't think it's possible to predict I think yeah I think what we already have is an incredibly powerful platform to solve lots of societally important problems that are currently unsolved so I just hope that people will lots of people will learn this toolkit and try to use it I don't think we need a lot of new technological breakthroughs to do a lot of great work right now and when do you think we're going to create a human level intelligence system do you think know how hard is it how far away are we don't know don't have no way to know I don't know like I don't know why people make predictions about this because there's no data and nothing to go on and the Senate that's right it's just like there's so many societally important problems to solve right now I just don't find it a really interesting question to even answer so in terms of societally important problems what's the problem well is within reached for it well I mean for example there are problems that AI creates right so most specifically labor force displacement is going to be huge and people keep making this frivolous econometrics argument of being like oh there's been other things that aren't AI that have come along before and haven't created massive labor force displacement therefore AI want it slow so there's a serious concern for you oh yeah Andrew yang is running on it yeah it's it's it's I'm desperately concerned and you see already that the changing workplace has lived to a hollowing out of the middle class you're seeing that students coming out of school today have a less rosy financial future ahead of them and the parents did which has never happened in recent in the last few hundred years you know we've always had progress before and you see this turning into anxiety and despair and and even violence so I very much worry about that quite a bit about ethics too I do think that every data scientist working with deep learning needs to recognize they have an incredibly high leverage tool that they're using that can influence society in lots of ways and if they're doing research that that research is going to be used by people doing this kind of work and they have a responsibility to consider the consequences and to think about things like how will humans be in the loop here how do we avoid runaway feedback loops how do we ensure an appeals process for humans that are impacted by my algorithm how do I ensure that the constraints of my algorithm are ethically explained to the people that end up using them there's all kinds of human issues which only data scientists are actually in the right place to educate people about but data scientists tend to think of themselves as just engineers and that they don't need to be part of that process just know yeah which is wrong well you're in the perfect position to educate them better to read literature to read history to learn from history well Jeremy thank you so much for everything you do for inspiring huge amount of people getting them into deep learning and having the ripple effects the the flap of a butterfly's wings that will probably change the world so thank you very much yes you