Vladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence | Lex Fridman Podcast #71
- The following is a conversation
with Vladimir Vapnik, part two, the second time
we spoke on the podcast. He's the co inventor of
support vector machines, support vector clustering, VC theory and many foundational ideas
in statistical learning. He was born in the Soviet Union, worked at the Institute of
Control Sciences in Moscow, then in the U.S., worked at ATT&T, NEC Labs, Facebook AI Research, and now is a professor
at Columbia University. His work has been cited
over 200,000 times. The first time we spoke on the podcast was just over a year ago,
one of the early episodes. This time we spoke after
a lecture he gave titled: "Complete Statistical Theory of Learning," as part of the MIT series
of lectures on Deep Learning and AI that I organized. I'll release the video of the
lecture in the next few days. This podcast and the lecture
are independent from each other so you don't need one
to understand the other. The lecture is quite
technical and math heavy. So if you do watch both,
I recommend listening to this podcast first, since
the podcast is probably a bit more accessible. This is The Artificial
Intelligence Podcast. If you enjoy it, subscribe on YouTube, give it five starts on Apple PodCast, support it on Patreon, or simply
connect with me on Twitter @LexFridman, spelled: F-R-I-D-M-A-N. As usual, I'll do one or
two minutes of ads now, and never any ads in the
middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance
app on the App Store. When you get it, use code: LexPodcast. Cash App lets you send money to friends by BitCoin and invest in the stock market with as little as $1. Broker services are provided
by Cash App Investing, a subsidiary of Square,
and member S.I.P.C.. Since Cash App allows you
to send and receive money digitally peer to peer,
and security in all digital transaction is very
important, let me mention that PCI data security standard. PCI DSS Level One, that
Cash App is complaint with. I'm a big fan of standards
for safety and security and PCI DSS is a good example of that. Where a bunch of competitors got together and agreed that there needs
to be a global standard around the security of transactions. Now we just need to do the
same for autonomous vehicles and A.I. systems in general. So again, if you get Cash
App from the App Store or Google Play, and use
the code: LexPodcast, you get $10, and Cash App
will also donate $10 to FIRST, one of my favorite
organizations that is helping to advance robotics and STEM education for young people around the world. And now, here's my conversation
with Vladimir Vapnik. You and I talked about
Alan Turing yesterday, a little bit.
- Yes. - And that he, as the father
of artificial intelligence may have instilled in our
field an ethic of engineering in that science. Seeking more to build intelligence rather than to understand it. What do you think is the
difference between these two paths of engineering intelligence and
the science of intelligence? - It's a completely different story. Engineering is imitation
of human activity. You have to make a device which behaves as a human behaves. You have all the functions of human. It does not matter how you do it. But to understand what
is intelligence about, is quite different problem. So I think, I believe, that it's somehow related to
predicated talk yesterday. Because, look at Vladimir Propp's idea. He just found such a one here, predicates. He called it units. Which can explain human behavior, at least in Russian tales. You look at the Russian
tales and derive from that. And then people realize that they're more violent in Russian tales. It is in TV, in movie
serials and so on and so on. - So you're talking about Vladimir Propp, who in
1928 published a book, "Morphology of the Folk Tale."
- Exactly. - Describing 31 predicates that have this kind of sequential structure that a lot of the
stories' narratives follow in Russian folklore and in other content. We'll talk about it; I'd
like to talk about predicates in a focused way, but let
me, if you'll allow me, to stay zoomed out on
our friend Allen Touring. And you know, he inspired a generation with the imitation game.
- Yes. - Do you think, if we can linger
on that a little bit longer do you think we can learn? Do you think learning
to imitate intelligence can get us closer to
understanding intelligence? Why do you think imitation
is so far from understanding? - I think that it is different between you have different goals. Your goal is to create
something, something useful. And that is great, and you can see how much things was done and I believe that it
will be done even more. Self-driving cars and all
sorts of this business. It is great, and it was
inspired by Turing's vision. But understanding is very difficult. It's more or less a
philosophical category. What means understandable? I believe in things
which start from Plato. That there exists world of ideas. I believe that intelligence,
it is world of ideas. But it is world of pure ideas. And when you combine that with reality things, it creates as in my case, in the variants, which is very specific. And that I believe, the combination of ideas and a way to
constructing the variant is intelligence, but
first of all a predicate. If you know predicate, and hope for this is not too much predicate exists. For example, sort of
unpredicted for human behavior is not a lot.
- Vladimir Propp used 31 (sighs) you could even call
'em predicates, 31 predicates to describe stories, narratives. Do you think human behavior, how much of human behavior,
how much of our world, our universe, all the things
that matter in our existence can be summarized in predicates of the kind that Propp was working with? - I think that we have a lot of forms of behavior. But I think the predicate is much less. Because even in these examples
which I gave you yesterday, you saw that predicate can be, one, predicate can construct
many different invariants, depending on your data. They're applying to different data and they give different invariants. But pure ideas, maybe not so much.
- Not so many. - I don't know about that. But my guess, I hope, that's my challenge
about digit recognition, how much you need. - I think we'll talk about computer vision and 2-D images a little
bit in your challenge. - [Vladimir] That's
exact about intelligence. - That's exactly about, no, that hopes to be exactly about
the spirit of intelligence in the simplest possible way.
- Yeah, absolutely. You should start the simplest way, otherwise you will not be able to do it. - There's an open question whether starting at the MNIST digit recognition is a step towards intelligence
or it's an entirely different thing.
- I think that to build records using say, 100,
200 times less examples, you need intelligence.
- You need intelligence. So let's, because you use this term, and it would be nice,
I'd like to ask simple, maybe even dumb questions. Let's start with a predicate. In terms of terms and
how you think about it, what is a predicate? - I don't know. (laughs)
I have a feeling, formally they exist. I believe that predicate for 2-D images. One of them is symmetry. - Hold on a second. Sorry, sorry to interrupt
and pull you back. At the simplest level, we're
not being profound currently. A predicate is a statement
of something that is true. - [Vladimir] Yes. - Do you think of predicates
as somehow probabilistic in nature or is this binary? This is truly constraints
of logical statements about the world.
- In my definition, the simplest predicate is function. Function and you can use this function to make inner product, that is predicate. - What's the input and what's
the output of the function? - Input is X, something
which is input in reality. Say, if you considered digit
recognition, in pixel space. But it is function which in pixel space. But it can be any function, pixel space. And you choose, and I believe that there are several functions, which is important for
understanding of images. One of them is symmetry, it's
not so simple construction, as I described is the
derivative, it's all this stuff. But another I believe,
I don't know how many, is how well-structurized is picture. - Structurized?
- Yeah. - What do you mean by structurized? - It is formal definition, say, something heavy on the left corner, not so heavy in the middle and so on. You describe in general,
concept of what you assume. - Concepts, some kind
of universal concepts. - Yeah. But I don't know how to formalize this. - This is the thing,
there's a million ways we can talk about this,
I'll keep bringing it up. We humans have such concepts, when we look at digits. But it's hard to put them,
just like you're saying now, it's hard to put them into words. - You know, that is example. When critics in music, trying to describe music, they use predicate. And not too many predicate
but in different combination. But they have some special
words for describing music. And the same should be for images. But maybe there are critics who understand essence of what this image is about. - Do you think there
exists critics who can summarize the essence
of images, human beings? - [Vladimir] I hope that, yes. - Explicitly state them on paper? The fundamental question I'm asking is do you (chuckles) do
you think there exists a small set of predicates
that will summarize images? It feels to our mind like it does that the concept of what
makes a two and a three and a four--
- No, no, no. It's not only on this level. It should not describe two, three, four. It describes some
construction which allows you to create an invariants. - And invariants, sorry to stick on this, but terminology.
- Invariants, it is, it is projection of your image. Say, I can say, looking at my image, it is more or less symmetric and I can give you a variable of symmetry. Say, level of symmetry using this function which I gave yesterday. Then you can describe that your image has these characteristics
exactly in the way how musical critics describe music. But this is invariant
applied to specific data, to specific music, to something. I strongly believe in this Plato idea, that exists world of
predicate and world of reality and predicate and reality
are somehow connected and you have to figure out that. - Let's talk about Plato a little bit. So you draw a line from Plato to Hegel to Wigner to today.
- Yes. - So Plato has forms. The theory of forms;
there's a world of ideas and a world of things, as you talk about. And there's a connection,
and presumably the world of ideas is very small,
and the world of things is arbitrarily big, but they're all, what Plato calls them like, it's a shadow, the real world is a shadow from the world of form.
- Yeah and you have projection - Projection.
- Of a world of idea. - Yeah, very poetic.
- And in reality you can, realize this projection
using these invariants because it is a projection
for only specific examples which create specific
features of specific objects. - So the essence of intelligence is, while only being able to
observe the world of things, try to come up with a world of ideas. - Exactly, like in this music story. Intelligent musical
critics knows this world and have a feeling about them.
- I feel like that's a contradiction;
intelligent music critics. I think music is to be... enjoyed in all its forms. The notion of critic, like a food critic. - [Vladimir] No, I don't
want attach emotion. - That's an interesting question. Does emotion, there's certain elements of the human psychology,
of the human experience which seem to almost contradict
intelligence and reason. Like emotion, like fear. Like love, all of those things. Are those not connected in
any way to the space of ideas? - This I don't know. I just want to be concentrating on very simple story. On digit recognition. - So you don't think you have to love and fear death in order
to recognize digits? - I don't know because
it's so complicated. It involves a lot of stuff
which I never considered. But I know about digit recognition. And I know that for digit recognition to get the records from small
numbers of observations, you need predicate but
not special predicate for this problem. But universal predicate which understand world of images. - Of visual information.
- Visual, yeah. But on the first step, they understand say, a world of 100 digits or characters or something simple. - [Lex] So like you said,
symmetry is an interesting one. - No, that's what I think
one of the predicate is related to symmetry,
the level of symmetry. - Okay, degree of symmetry.
- Yeah. - So you think symmetry at the bottom is a universal notion and there's, there's degrees of a
single kind of symmetry? Or is there many kinds of symmetries? - Many kinds of symmetries. There is a symmetry/anti-symmetry
say, letter S. So it has vertical anti-symmetry. It could be diagonal
symmetry, vertical symmetry. - So when you cut
vertically the letter S-- - Yeah, and then the upper part and lower part
are in different directions. - Inverted along the Y-axis. But that's just like, one
example of symmetry right? Isn't there like--
- Right, but there is a degree of symmetry. If you play all this lineated stuff to do tangent distance, whatever I described, you can have a degree of symmetry. And that is what is
describing reason of image. It is the same as you will describe this image saying about digit S has anti-symmetry, digit three is symmetric more/less. Look for symmetry. - Do you think such concepts
like symmetry predicates, like symmetry, is it a
hierarchical set of concepts? Or are these independent, distinct predicates that we want to discover a subset of? - There is a degree of symmetry. And this idea of symmetry made very general, like degree of symmetry, the degree of symmetry can
be zero, no symmetry at all. Or degree of symmetry of let's say, more or less symmetrical. But you have one of these descriptions, and symmetry can be different. As I told, horizontal, vertical, diagonal. Anti-symmetry also is
a concept of symmetry. - What about shape in general? I mean, symmetry is a fascinating notion, but it--
- No, no, I'm talking about digits, I would like to concentrate on all I would like to know
predicate for digit recognition. - Yes, but symmetry is not
enough for digit recognition, right?
- It is not necessarily for digit recognition; it helps to create invariant, which you can use when
you will have examples for digit recognition. You have regular problem
of digit recognition. You have examples of the
first class and second class. Plus you know that there exists
this concept of symmetry. And you apply when you're
looking for decision rule, you will apply concept of symmetry, of this level of symmetry
which you estimate from. Everything is, comes from weak convergence. - What is convergence? What is weak convergence? What is strong convergence? I'm sorry I'm gonna do this to you. What are we converging from and to? - You're converging, you
would like to have a function. The function which say, indicate
a function which indicate your digit five, for example.
- A classification/task-- - Let's talk only about
classification task. - So classification means you will say whether this is a five or not, or say which of the 10 digits it is. - Right, right, I would like to have these functions. Then, I have some examples. I can consider property of these examples. Say, symmetry, and I can
measure level of symmetry for every digit. And then I can take average from my training data. And I will consider only functions of conditional probability,
which I'm looking for in my decision level. Which applying to the digits will give me the same average as I observed on training data. So actually this is different level of description of what you want. You want, not just, you show not one digit. You show this predicate,
show general property of all digits which you have in mind. If you have in mind digit three, it gives you property of digit three and you select as
admissible set of functions, only function which keeps this property. You will not consider the other functions. So you're immediately
looking for smaller subset of functions.
- That's what you mean by an admissible function.
- And admissible function. Exactly.
- Which is still a pretty large, for the number three, that's a large--
- It's large but, if you have one predicate. But according to, there is a strong and weak convergence. Strong convergence if
convergence in function. You're looking for the
function, on one function, and you're looking on another function. And square difference from them should be small. If you take difference at any points, make a square, make an integral,
and it should be small. That is convergence in function. Suppose you had some
function, any function. So I would say, I say that some function converged to this function. If integral from square
difference between them is small-- - That's the definition
of strong convergence. - That's the definition of--
- Two functions, the integral of the difference is small.
- Yeah, it is convergence in functions.
- Yeah. - But you have different
convergence in functions. You take any function,
you take some function Fe and take inner product this
function, this F function. F zero function which you want to find, and that gives you some value. So you say that a set of functions converge in inner product to this function if this value of inner product
converge to value F zero. That is for one Fe. But three converges requires that it converges for any
function of field of space. If it converge for any
function of field of space, then you will say that
this is a weak convergence. You can think that when you take integral, that is property, integral
property of function. For example, if you will
take sine or cosine, it is coefficient of say, free expansion. If it converged for all coefficients or free expansion, so under some condition it converged to function you're looking for. But weak convergence means any property. Not convergence, not point-wise. But integral property of function. So weak convergence means
integral property of functions. When I'm talking about
predicate, I would like to formulate which integral properties I would like to have for convergence. And if I will take one predicate, predicate its function
which I measure property. If I will use one predicate and say, I will consider only
function which give me the same value as this
predicate, I select a set of functions from functions
which is admissible, in the sense that
function which I'm looking for in this set of functions. Because I'm checking in training data, it gives the same. - [Lex] Yeah, so it
always has to be connected to the training data in terms of-- - Yeah, but property, you can know independent of training data. And this guy, Propp, says that there is formal property. 31 property and if you--
- A fairy tale, the Russian fairy tale.
- Right. But the Russian fairy tale
is not so interesting. It's more interesting
that people applied this to movies, to theater, to different things and the same works. They're universal. - So I would argue that
there's a little bit of a difference between the kinds of things that would apply to, which are essentially stories
and digit recognition. - [Vladimir] It is the same story. - You're saying digits, there's
a story within the digit? - Yeah.
(laughing) So but my point is, well, I hope that it's
possible to beat a record. Using not 60,000 but say, 100 times less. Because instead you will give predicates. And you will select your decision not from right set of functions. But from set of function
which gives this predicate. But predicate is not related
just to digit recognition. - Right, so--
- Like in Plato's case. - (laughs) Do you think it's possible to automatically discover the predicates? So, you basically said that the essence of intelligence is the discovery of good predicates. - [Vladimir] Yeah. - Now the natural question is, you know, that's what Einstein
was good at doing in physics. Can we make machines do these kinds of discovery of good predicates? Or is this ultimately a human endeavor? - That I don't know. I don't think that much it can do. Because, according to theory
about weak convergence, any function from Hilbert
Space can be predicate. So you have infinite number
of predicate in upper. And before you don't know which
predicate is good in which. But whatever Propp showed, and why
people call it breakthrough, that there is not too many predicate which cover most of situation
that happen in the world. - So there's a sea of predicates. Only a small amount are
useful for the kinds of things that happen in the world? - I think that, I would say only a small part of predicates very useful. Useful, all of them.
- Only a very few are what we should, let's
call them good predicates. - Very good predicates.
- Very good predicates. Can we linger on it,
what's your intuition? Why is it hard for a machine to
discover good predicates? - Even in my talk described
how to do a predicate. How to find new predicates, I'm not sure that it is very good predicate. - [Lex] What did you propose in your talk? - In my talk I gave example for diabetes. - Diabetes, yeah.
- When we achieve some percent so then we're looking from area, where some sort of predicate
which I formulated, does not... keep invariant. So if it doesn't keep, I retrain my data. I select only functions
which keep this invariant. In the way I did it, I
improved my performance. I came looking for this predicate. I know technically how to do that. You can of course do it using a machine, but I'm not sure that we will construct
the smartest predicate. - Well this is the,
allow me to linger on it. Because that's the essence,
that's the challenge. That is artificial, that's
the human-level intelligence that we seek, is the discovery
of these good predicates. You've talked about deep
learning as a way to, the predicates they use and
the functions are mediocre. Or you can find better ones. - Let's talk about deep learning. - Sure, let's do it.
- I know only Jans LaComb, convolutional network, and what else? And it's very simple convergence. - [Lex] There's not much else to know. - To fix it left and right.
- Yes. - I can do it like that
with one predicate. - [Lex] Convolution is a single predicate. - It's single, it's single predicate.
- Yes, but-- - You know exactly,
you take the derivative for translational, and
predicate should be kept. - So that's a single predicate,
but humans discovered that one or at least--
- Not that that is the least not too many predicates. And that is big story because
Jan did it 25 years ago, and nothing so clear was
uttered to deep network. And then I don't understand why we should talk about deep network instead of talking about
piece-wise linear functions which keeps this predicate. - You know, a counter argument is, that maybe the amount
of predicates necessary to solve general intelligence,
say in the space of images, doing efficient recognition
of handwritten digits is very small. So we shouldn't be so
obsessed about finding, we'll find other good predicates like convolution, for example. There has been other advancements like, if you look at the work with attention, there's attentional mechanisms, especially used in natural language focusing the network's ability to, to learn at which part
of the input to look at. The thing is, there's other
things besides predicates that are important for the
actual engineering mechanism of showing how much you can really do, given such these predicates. I mean, that's essentially
the work of deep learning is constructing
architectures that are able to be given the training data
to be able to converge towards a function that can approximate, that can generalize well. It's an engineering problem.
- Yeah, I understand. But let's talk not on an emotional level,
but on a mathematical level. You have set of piece-wise
linear functions, it is all possible neural networks. It's just piece-wise linear functions. It's many, many pieces.
- Large, large number of piece-wise linear function.
- Exactly. - Very large.
- Very large. - [Lex] Almost, feels like too large. - It's still simpler
than say, convolution, which is reproducing Hilbert's Space and we have a Hilbert's set of functions. - What's Hilbert Space? - It's space with infinite
number of coordinates. A function for expansion,
something like that. So it's much richer. And when I'm talking about
closed-form solution, I'm talking about this set of functions. Not piece-wise linear set,
which is particular case of (chuckles) it is small part of it. - So neural networks is a small part of the space you're, of
functions you're talking about? - Small set of functions.
- Yeah. - Let me take it that. But it is fine, it is fine. I don't want to discuss the
small or big we take and what. So you have some set of functions. So now when you're trying
to create architecture, you would like to create
admissible set of functions, which all your tricks,
to use not all functions, but some subset of this set of functions. Say when you're introducing
convolutional network. It is a way to make this subset useful for you. But from my point of view, convolutional, it is something, you want
to keep some invariance. Say, translation invariance. But now if you understand this, and you cannot explain on the level of ideas
what neural network does, you should agree that it is much better to
have a set of functions. As I say, this set of
functions should be admissible, it must keep this
invariant, this invariant, and that invariant. You know that as soon as you
incorporate new invariants, set of functions becomes
smaller and smaller and smaller. - [Lex] But all the invariants are specified by you the human. - Yeah, but what I am hope, that there is a standard
predicate like, Propp showed. That's what I want to find
for digit recognition. If we start, it is completely new area of what is intelligence
about on the level, starting from Plato's idea. What is the world of ideas? And I believe there is not too many. But you know, it is
amusing that mathematicians are doing something with neural
network in general function. But people from literature, from art, they use this all the time. - That's right.
- Invariants. Say, it is great how
people describe music. We should learn from that. Something on this level. So why, Vladimir Propp
who was just theoretical. Who studied theoretical
literature, he found that. - You know what, let me
throw that right back at you. Because there's a little bit of a, that's less mathematical
and more emotional, philosophical Vladimir Propp. I mean, he wasn't doing math.
- No. - And you just said another
emotional statement, which is you believe that this Plato world of ideas is small. - I hope.
- I hope. Do (chuckles), do you, what's your intuition, though,
if we can linger on it? That bothers me.
- You know, because not just small or big. I know exactly. That when I'm introducing some predicate, I
decrease set of functions. But my goal to decrease
set of functions much. - By as much as possible.
- By as much as possible. Good predicate which does this, then I should choose next
predicate which does this, which decreases set as much as possible. So, set of good predicates, it is such that the decrease, amount of admissible functions-- - So if each good predicate
significantly reduces the set of admissible functions that there naturally should
not be that many predicates. - No, but, if you reduce very well the
VC dimension of the function of admissible set of
functions, it's small. And you need not too much
training data to do well. - [Lex] And VC dimension, by the way, is some measure of capacity
of this set of functions. - Right, roughly speaking,
how many function in this set. So you're decreasing, decreasing, and it makes it easier for you to find the function you're looking for. But the most important part, to create a good admissible
set of functions. And it probably is that
there are many ways but, a good predicate is such
that it can do that. For this duck, you should know a little bit about duck. - What are the three
fundamental laws of ducks? - Looks like a duck, swims like a duck, and quacks like a duck.
- And quacks. You should know something about ducks to be able to--
- Not necessarily. Looks like say, horse. It's also good. - [Lex] So it's not (chuckles), it generalize from ducks.
- Yes, and talk like it, and make sound like horse or something. And run like horse and moves like horse. It is general, it is general predicate that this applies to duck. But for duck you can say,
play chess like duck. - [Lex] You cannot say,
play chess like a duck. - Why not?
- So you're saying you can but that that would not be a good-- - [Vladimir] No, you will
not reduce that function. - Yeah, you would not
reduce the set of functions. - So you can, the story is formal story and it's a magical story
is that you can use any function you want as a predicate. But some of them are good,
some of them are not. Because some of them
reduce a lot of functions. The admissible set, some of them (mumbles) - So the question is, and
I'll probably keep asking this question, but how do
we find such predicates? What's your intuition? Handwritten recognition, how do we find the answer to your challenge? - Yeah, I understand it like that. I understand what. - What defined?
- What that means, a new predicate.
- Yeah. - Like, guy who understands music can say these words he describes when he listens to music. He understands music, he'll
use not too many different. Or you can do it like Propp. You can make collection, what
you're talking about music. About this, about that. It's not too many different
situations you describe. - Because we mentioned
Vladimir Propp a bunch, let me just mention, so
there's a sequence of 31 structural notions that
are common in stories. And I think--
- They're called units. - Units, and I think they resonate. It starts, just to give an example, Absention: a member of
the hero's community or family leaves the security
of the home environment, then goes through the interdiction, a forbidding edict or command
that's passed upon the hero. Don't go there; don't do this. The hero's warned against some action. Then step three:
Violation of interdiction. You know, break the rules,
break out on your own. Then reconnaissance, the
villain makes an effort to attain knowledge needed
to fulfill their plot, so on. It goes on like this. Ends in a wedding, number
31, happily every after. - He just gave description
of all situations. He understands his world--
- Of folk tales. - Yeah, not folk--
- Stories. - Stories, and these stories
not just in folk tales. These stories in
detective serials as well. - [Lex] And probably in our lives. We probably live--
- Read this. They're all, this predicate is good
for different situations. For movie, for theater.
- By the way, there's also criticism, right? There's another way to
interpret narratives. From... Claude Levi Strauss.
- I am not in this business. - I know, that's theoretical literature, but it's looking at paradigms behind them. - [Vladimir] It's
always, this discussion-- - Philosophers argue.
- Yeah. - Yeah.
- But at least there is units. It's not too many units that can describe. But describe probably
gives the other units. Or another way for description. - Exactly, another set of units. - [Vladimir] Another
set of predicates, yes. It doesn't matter how,
but they exist probably. - My question is, whether
given those units, whether without our human
brains to interpret these units, they would still hold as
much power as they have. Meaning, are those units
enough when we give them to the alien species? - Let me ask you, do you understand digit images? - No, I don't understand.
- No, no, no. When you can recognize these digit images, it means that you understand. - Yes, I understand.
- You understand characters, you understand.
- Nope, nope, nope, nope. It's the imitation versus
understanding question. Because I don't understand the mechanism by which I understand--
- No, no. I'm not talking about, I'm
talking about predicates. You understand that it involves symmetry, maybe structure, maybe something else. I cannot formulate, I just was able to find symmetries, or
degree of symmetries. - So this is a good line. I feel like I understand
the basic elements of what makes a good hand
recognition system my own. Like, symmetry connects with me. It seems like that's a
very powerful predicate. My question is, is there
a lot more going on that we're not able to introspect? Maybe I need to be able to understand a huge amount in the world of ideas. Thousands of predicates,
millions of predicates in order to do hand recognition. - [Vladimir] I don't think so. - Both your hope and your intuition are such that very few--
- No, no, let me explain. You're using digits. You're using examples as well. Theory says that if you will use all possible functions from Hilbert Space, all possible predicates, you
don't need training data. You just will have
admissible set of functions which contain one function.
- Yes. So the trade off is, when
you're not using all predicates, you're only using a few good predicates, that you need to have some training data. - Yes, exactly.
- The more good predicates you have, the less training data you need.
- Exactly. That is intelligent learning. - Okay, I'm gonna keep asking
the same dumb question, handwritten recognition,
to solve the challenge, you kind of propose a challenge that says we should be able to get state
of the art MNIST error rates by using very few, 60 maybe
few examples per digit. What kind of predicates
do you think you'll-- - [Vladimir] That is the challenge. (laughs) So people who will solve this problem. - They will answer.
- They will answer it. - Do you think they'll
be able to answer it in a human explainable way? - They just need the
right function, that's it. - But so, can that
function be written I guess by an automated reasoning system? Whether we're talking
about a neural network learning a particular
function, or another mechanism. - No, I'm not against neural network. I am against admissible set of function which creates neural network. You did it by hand. You don't do it by invariants, by predicate, by reason. - But neural networks can then reverse, to the reverse step of
helping you find a function. The task of a neural network is to find disentangled representation,
for example is what they call, is to fine that one predicate function that's really captures
some kind of essence. Not the entire essence,
but one very useful essence of this particular visual space. Do you think that's possible? Listen, I'm grasping, hoping
there's an automated way to find good predicates, right? So the question is,
what are the mechanisms of finding good predicates, ideas, that you think we should pursue? A young grad student listening right now. - I gave example. So find situation where predicate, which you're suggesting, don't create invariant. It's like in physics, find situation where existing theory cannot explain it. - [Lex] Find a situation
where the existing theory can't explain it.
- Cannot explain this. - [Lex] So you're finding contradictions. - Find contradiction, and then
remove this contradiction. But in my case, what means contradiction, you find function which, if
you will use this function, you're not keeping invariants. - [Lex] So really the process of discovering contradictions. - Yeah. It is like in physics. Find situation where
you have contradiction for one of the property, for one of the predicate, then include this predicate
making invariants. And solve, again, this problem, now you don't have contradiction. But it is not the best way probably, I don't know, to looking for predicate.
- That's just one way. Okay.
- That, no, no. It is brute force way.
- The brute force way. What about the ideas of what, big
umbrella term of symbolic AI? In the '80s with Xper Systems, sort of logic, reasoning-based systems. Is there hope there to find some, through sort of, deductive reasoning, to find good predicates? - [Vladimir] I don't think so. I think that just logic is not enough. - Kind of a compelling notion, though. That when smart people sit in a room and reason through things,
it seems compelling. And making our machines do
the same is also compelling. - Everything is very simple when you have infinite
number of predicate. You can choose the function you want. You have invariants and you
choose the function you want. But you have to have not too many invariants to solve the problem. And how from infinite number function to select finite number and hopefully small finite
number of functions. Which is good enough to extract some small set
of admissible functions. So they will be admissible, it's for sure, because every function just
decrease set of function and leaving it admissible. But it will be small. - But why do you think
logic-based systems can't help? Intuition, not--
- Because you should know reality, you should know life. This guy like Propp, he knows something and he tried to put in
invariants his understanding. - So but that's the human, yeah, yeah. But see, you're putting too much value into Vladimir Propp's knowing something. - No, it is--
- Am I being misunderstanding? - What means you know life? What it mean?
- You know common sense. - No, no, you know something. Common sense, it is some rules. - You think so? Common sense is simply rules? Common sense is every, it's mortality, it's fear of death, it's love, it's spirituality, it's happiness and sadness. All of it is tied up into
understanding gravity which is what we think of as common sense. - I don't credit or discuss of that. I want to discuss understand
digit recognition. - Any time I bring up love and death you bring it back to digit recognition. I like it. (laughs)
- No, you know, it is doable because there is a challenge. - Yeah.
- Which I still have to solve it; if I will
have a student concentrate on this work, I will
suggest something or so. - You mean handwritten recognition? Yeah, it's a beautifully simple, elegant, and yet--
- I think that I know invariants which will solve this. - You do?
- I think that, I think that. But it is not universal. I want some universal
invariants which are good not only for digit recognition,
for image understanding. - So let me ask, how hard do you think is 2-D image understanding? If we can kind of intuit
handwritten recognition, how big of a step, leap,
journey is it from that? If I gave you good, if
I solved your challenge for handwritten recognition,
how long would my journey then be from that to
understanding more general, natural images?
- Immediately. You will understand it as soon
as you will make a record. - You think so?
- Because it is not for free. As soon as you will
create several invariants which will help you to get the same performance that the best neural net did using more than 100 times less examples. You have to have something
smart to do that. - And you're saying?
- That's not an invariant. It is predicate because
you should put some idea how to do that. - But okay. Let me just pause, maybe it's
a trivial point, maybe not, but handwritten recognition
feels like a 2-D, two-dimensional problem. And it seems, like how much complicated is the fact that most
images are a projection of a three-dimensional
world onto a 2-D plane? It feels like for a
three-dimensional world, we need to start
understanding common sense in order to understand an image. It's no longer visual shape and symmetry. It's having to start to
understand concepts of, understand life.
- Yeah. You're talking that there
are different invariant, different predicate, yeah. - [Lex] And potentially
much larger number. - You know, maybe, but
let's start from simple. - [Lex] But you said that
it would be immediate. - No, you know, I cannot
think about things which I don't understand yet. This I understand. But I'm sure that I don't
understand everything there. - [Lex] Yeah, that's the difference-- - It's like they say, do as simple as possible, but not simpler, and that is exact case. - With handwritten rec--
- With handwritten. - Yeah, but that's the
difference between you and I. (laughs) I welcome and enjoy thinking about things that I
completely don't understand. Because to me it's a natural extension, without having solved
handwritten recognition to wonder how, how difficult is the next
step of understanding 2-D and 3-D images, because ultimately, while the science of
intelligence is fascinating, it's also fascinating to see how that maps to the
engineering of intelligence. And recognizing handwritten digits is not, doesn't help you, it might, it may not, help you with the problem
of general intelligence. We don't know; it'll
help you a little bit. We don't know how much.
- It is unclear. - It's unclear.
- Yeah. - It might very much.
- But I would like to make a remark: I start not from very primitive problem like, challenge problem; I start
with very general problem, with Plato, so you understand. And it comes from Plato,
digit recognition. - So you basically took Plato and the world of forms and ideas, and mapped, and projected
it into the clearest, simplest formulation of that big world into handwritten recognition. - I will say that I did
not understand Plato until recently, and until I considered weak convergence
and then predicate and then, oh, this is what Plato thought. - Can you linger on that? How do you think about this world of ideas and world of things and Plato? - It is a metaphor.
- It's a metaphor for sure. It's a compelling, it's a
poetic and a beautiful metaphor. But what can you--
- But it is a way how you should try to understand how a duck ideas in the world. So from my point of view, it is very clear, but it
is a line all the time people looking for that. Say, Plato then Hegel, whatever reasonable it exists, whatever exists it is reasonable. I don't know what he
have in mind, reasonable. - [Lex] Right, these philosophers again. - No, no, no, no, no, no, it is, it is next stop of Wilner. That which we might understand
something of reality. He did the same Plato line. And then it comes suddenly
to Vladimir Propp. Look, 31 ideas, 31 units,
and describes everything. - There's abstractions, ideas
that represent our world, and we should always
try to reach into that. - Yeah, but what you
should make a projection on the reality, but understanding is, it is abstract ideas. You have in your mind
several abstract ideas which you can apply to reality. - And reality in this case so
if we look at machine learning is data.
- This example, data. - Data. Okay, let me put this on you, because I'm an emotional creature. I'm not a mathematical creature like you. I find compelling the
idea, forget the space, this sea of functions. There's also a sea of data in the world. And I find compelling that there might be, like you said, teacher, small examples of data
that are most useful for discovering good,
whether it's predicates or good functions, that
the selection of data may be a powerful journey,
a useful mechanism. You know, coming up with
a mechanism for selecting good data might be useful, too. Do you find this idea of
finding the right data set interesting at all? Or do you kinda take
the data set as a given? - I think that it is, you know, my thing is very simple. You have huge set of functions. If you will apply, and you
have not too many data. If you pick up function
which describes this data, you will do not very well. - Like randomly pick?
- Yeah (mumbles) It will be irritating. So you should decrease set of function from which you're picking out one. So you should go somehow to
admissible set of functions. And this, what about weak convergence? From another point of view, to make admissible set of function,
you need just the deal, just function which you
will take an inner product. Which you will measure property of your function. That is how it works. - [Lex] No, I get it, I get it. I understand it but do you, the reality is--
- But let's discuss, let's think about examples. You have huge set of function
and you have several examples. If you just trying to keep, take function which
satisfies these examples, you still will not have it. You need decrease, you need
admissible set of functions. - Absolutely, but what, say you have more data than functions. I mean, maybe not more
data than functions, 'cause that's--
- That's impossible. - Impossible, I was trying
to be poetic for a second. I mean, you have a huge amount of data, a huge amount of examples. - But amount of function can be even bigger.
- Even bigger, I understand. - Everything is (chuckles) - [Lex] There's always a bigger boat. - Whole Hilbert Space of functions. - I got you, but okay. But you don't, you don't
find the world of data to be an interesting optimization space? Like, the optimization should
be in the space of functions. - In creating admissible set of functions. - [Lex] Admissible set of functions. - You know, even from the
classical, this is so. From structured reasoning,
you should organize function in the way they will be useful for you.
- Right. - And that is admissible step. - But the way you're thinking about useful is you're given a small set of examples. - Small set of functions which contain functions by looking for it. - Yeah, but as looking for
it based on the empirical set of small examples.
- Yeah. But that is another
story, I don't touch it. Because I believe that
these small examples, it's not too small. Say 65%, law of large numbers works. I don't need uniform law. The story is that in
statistics there are two laws. Law of large numbers and
uniform law of large numbers. So I want a situation where
I use law of large numbers but not uniform law of large numbers. - [Lex] Right, so 60 is law of large. It's large enough.
- I hope, I hope. It still needs some evaluation, some balance, et cetera. What I did is the following that, if you trust that say, this average gives
you something close to expectation so you can talk about that, about this predicate.
- Yeah. - [Vladimir] And that is
basis of human intelligence. - Good predicates, the
discovery of good predicates is the basis of human intelligence. - No, no, it is discovery of your understanding of world. Of your total logic of
understanding the world. Because you have several functions which you will apply to reality. - Can you say that again? So you're--
- You have several functions, predicates, but they're abstract. Then you will apply them
to reality, to your data, and you will create in
this weak predicate, which is useful for your task. But predicate are not related specifically to your task, to this here task. It is abstract functions,
which being applied, applied to--
- Many tasks that you might be interested in.
- It might be many tasks. I don't know. Different tasks. - [Lex] Well they should be many tasks. Right?
- Yeah, I believe. Like in Propp case. It was for fairy tales, but
it's happened everywhere. - Okay, we talked about
images a little bit but, can we talk about Noam
Chomsky for a second? (laughing) - I don't know him very well. - Personally?
- Not personally I don't know. - His ideas.
- His ideas. - Well let me just say,
do you think language, human language is essential
to expressing ideas, as Noam Chomsky believes? So like, language is at the core of our formation of predicates. Human language. - In all the story of
language is very complicated. I don't understand this
and I thought about-- - Nobody does.
- I'm not ready to work on that because it's so huge. It is not for me, and I
believe not for our century. - The 21st century.
- Not for 21st century. We should learn something, a lot of stuff from simple tasks like digit recognition. - So you think digital recognition, 2-D image, how would you more abstractly define it, digit recognition? It's 2-D image, symbol recognition essentially? I'm trying to get a sense,
sort of thinking about it now, having worked with MNIST forever, how small of a subset is
this of the general vision recognition problem and the
general intelligence problem? Is it? Is it a giant subset, is it not? And how far away is language? - You know, let me refer to Einstein. Take the simplest problem,
as simple as possible, but not simpler, and this is
challenge, is simple problem. But it's simple by idea,
but not simple to get it. When you will do this, you will find some
predicate which helps it. - Yeah, with Einstein you can, you look at General Relativity, but that doesn't help you
with quantum mechanics. - And that's another story. You don't have any universal instrument. - Yeah, so I'm trying to wonder if which space we're in. Whether handwritten recognition
is like General Relativity and then language is
like, quantum mechanics, are you still gonna
have to do a lot of mess to universalize it but, I'm trying to see. What's your intuition why
handwritten recognition is easier than language? Just, I think a lot of
people would agree with that, but if you could elucidate sort of, the intuition of why. - I don't know, no, I don't think in this direction. I just think in the direction
that this is problem, which if you will solve it well, we will create some abstract understanding of images. Maybe not all images. I would like talk to guys who doing in real images in Columbia University. - What kind of images? Unreal you said? - Real images.
- Real images. - Yeah, what their idea is, there are predicate what can be predicate. I say symmetry will play a
role in real-life images. In any real-life images, 2-D images, let's talk about 2-D images. Because... that's what we know. And neural network was
created for 2-D images. - So the people I know in
vision science, for example, for people who study human vision, that they usually go
to the world of symbols and like, handwritten
recognition but not really. It's other kinds of symbols to study our visual perception system. As far as I know, not much
predicate-type of thinking is understood about our vision system. - [Vladimir] They did not
think in this direction. - They don't, yeah, but
how do you even begin to think in that direction? - That is, I would like to discuss this. Because if you'll be
able to show that it is what's working, and theoretical thing, it's not so bad. - So if we compare it to language, language has like letters,
a finite set of letters and a finite set of ways
that you can put together those letters so it feels more
amenable to kind of analysis. With natural images,
there is so many pixels-- - No, no, no, letter, language is much, much more complicated. It involves a lot of different stuff. It's not just understanding of simple class of tasks. I would like to see list of task where language is involved.
- Yes. So there's a lot of nice benchmarks now in natural language processing,
from the very trivial, like, understanding the
elements of a sentence, to question/answering,
so much more complicated where you talk about open domain dialogue. The natural question is,
will handwritten recognition, it's really the first step of understanding visual information. - Right. But even our records show that we're going wrong direction. Because we need 60,000 digits. - So even this first step, so forget about talking about the full journey. This first step should be
taking in the right direction. - No, no, in wrong direction
because 60,000 is unacceptable. - No, I'm saying it should be
taken in the right direction because 60,000 is not acceptable. - You can talk, it's great
we have 1/2 percent of error. - And hopefully the step from doing hand recognition
using very few examples, a step towards what
babies do when they crawl and they understand their
physical environment. - I don't know what baby do.
- I know you don't know about babies, but--
- If you will do from very small examples, you will find principals which are different.
- That will apply to babies. - From what we're using now. Theoretical it's more or less clear. That means you will use weak convergence, not just strong convergence. - Do you think these principals are, will naturally be human interpretable? - [Vladimir] Oh yeah. - So we'll be able to explain them and have a nice presentation to show what those principals are? Or are they very, going to be very kind of,
abstract kinds of functions? - For example, I talk
yesterday about symmetry. And I gave very simple examples. The same will be like that. - You gave like, a
predicate of a basic for-- - For symmetries.
- Yes. For different symmetries
and you have for-- - For degree of symmetries. That is important, not just symmetry exist and does not exist;
degree of symmetry. - [Lex] Yeah, for handwritten recognition. - It's not for handwritten,
it's for any images. But I would like to
apply it to handwritten. - Right, in theory it's more general. Okay, okay. So a lot of the things
we've been talking about falls, we've been talking
about philosophy a little bit, but also about mathematics and statistics. A lot of it falls into this idea, a universal idea of
statistical theory of learning. What is the most beautiful and sort of, powerful or essential idea
that you've come across, even for yourself just personally, in the world of statistics or
statistic theory of learning? - Probably uniform convergence
which we do with (mumbles). - [Lex] Can you describe
universal convergence? - You have law of large numbers. So for any function,
expectation of function, average of function converged expectation. But if you have a set of functions, for any function it is true. But it should converge similar to anywhere therefore all set of functions. For learning, you need uniform convergence; just
convergence is not enough. Because when you pick up
one which gives minimum, you can pick up one function which does not converge and it
will give you the best answer for this function. So you need the uniform
convergence to guarantee learning. Learning does not really enter
your law of large numbers, really universal. The idea of universal
convergence exists in statistics for a long time. It is interesting that as I think about myself, how stupid I was for 50 years, I did not see weak convergence. I work only on strong convergence. But now I think that most
powerful is weak convergence because it makes admissible
set of functions. And even in old proverbs, when people tried to understand
recognition about duck law, looks like a duck and so on, they used weak convergence. People in language they understand this. But when we're trying to
create artificial intelligence if we want invent in different way. Just consider strong convergence,
artificial intelligence. - So reducing the set
of admissible functions you think there should be effort put into
understanding the properties of weak convergence?
- You know, in classical mathematics,
in Hilbert Space, there are only two forms of
convergence, strong and weak. Now we can use both. That means that we did everything. And it so happened, that when we used Hilbert
Space, which is very rich space, space of continuous functions which has an integral and square. So we can apply weak and
strong convergence for learning and have closed-form solution. So for computation it is simple. For me it is sign that it is right way. Because you don't need any this theory. Yes, do whatever you want. But now the only way left is the concept of what is predicate? - Of predicate.
- But it is not statistics. - By the way, I like
the fact that you think that heuristics are a mess
that should be removed from the system, so closed-form solution is the ultimate goal.
- No it so happens that when you're using right instrument, you
have closed-form solution. - Do you think intelligence,
human-level intelligence, when we create it will, will have something like
a closed-form solution? - Now I'm looking on bounds, which I gave bounds for convergence. And when I'm looking for bounds, I'm thinking, what is the most appropriate
kernel of this bound would be. So we know that in say, all our businesses we use
radial basis function. But looking for the bound I think that I start to understand
that maybe we need to make corrections to
radial basis function, to be closer to what's better for these bounds. So I'm again trying to
understand what type of kernel have best approximation, not an approximation, best fit to these bounds. - Sure, so there's a
lot of interesting work that could be done in
discovering better function than the radial basis functions for the kinds of bounds you would find. - It still comes from, you're looking to match
and trying to understand. - From your own mind looking at the-- - Yeah but--
- I don't know. - Then I'm trying to understand what will be good for that. - Yeah, but to me there's still a beauty, again, maybe I'm descending
value toward heuristics. To me, ultimately intelligence
will be a mess of heuristics. And that's the engineering answer, I guess.
- Absolutely. When you're doing say, self-driving cars, the great guy who will do this. It doesn't matter what theory behind that. Who has a better theory have to apply. It is the same story about predicate because you cannot create
rule for, situation is much more than you have rule for that. Maybe you can have more abstract rules, then it will be less literal. It is the same story about ideas and ideas applied to specific cases. - But still you should--
- You cannot avoid this. - [Lex] Yes of course,
but you should still reach for the ideas to understand the science.
- Yeah, yeah. - Let me kind of ask, do you think neural networks or functions can be made to reason? What do you think, we've been
talking about intelligence, but this idea of reasoning. There's an element of
sequentially disassembling, interpreting the images. When you think of handwritten recognition, we kind of think that
there will be a single, there's an input and an output;
there's not a recurrence. - Yeah.
- What do you think about, sort of, the idea of recurrence? Of going back to memory
and thinking through this sort of, sequentially, mangling the different
representations over and over until you arrive at a conclusion? Or is ultimately all of
that can be wrapped up in a function? (chuckles) - You're suggesting, that let us use this type of algorithm. When I started thinking, I first of all, starting to understand what I want. Can I write down what I want? And then I try to formalize. And when I do that, I'm thinking
how to solve this problem. Till now I did not see situation where-- - Where you need recurrence.
- Recurrent. - But do you observe human beings? - Yeah.
- Do you try to, it's the imitation question, right? It seems that human beings reason, this kind of sequentially, sort of, does that
inspire in you a thought that we need to add that into our intelligence systems? You're saying, okay, you've kind of answered saying, until now I haven't seen a need for it. And so because of that,
you don't see a reason to think about it? - You know, most of
things I don't understand. In reasoning, in humans, it is for me too complicated. For me, the most difficult part is to ask questions, good questions. How it works, how people asking questions. I don't know this. - You said that machine
learning's not only about technical things,
speaking of questions, but it's also about philosophy. What role does philosophy
play in machine learning? We talked about Plato,
but generally thinking in this philosophical way, how does philosophy and math
fit together in your mind? - Just ideas, and then
their implementation. It's like predicate, say, admissible set of functions. It comes together, everything. Because, the first declaration of
theory was done 50 years ago, all that necessary, so everything there. If you have data you can, and you, in your set of functions has not big capacity. So law of inter-dimension,
you can do that. You can make structuralist
minimization, control capacity. But there was not table to make
admissible set of function with. Now when suddenly we realize that we did not use another idea
of convergence, which we can, everything comes together. - But those are mathematical notions. Philosophy plays a role of simply saying that we should be swimming
in the space of ideas. - Let's talk, what is philosophy? Philosophy means understanding of life. Understanding of life,
say people like Plato, they understand on very
high, abstract level of life. And whatever I'm doing, it's just implementation of
my understanding of life. But every new step, that is very difficult. For example, to find this idea that we need weak convergence, was not simple for me. - So that required thinking
about life a little bit. Hard to trace, but there
was some thought process. - You know, when I'm thinking
about the same problem for 50 years now, and again and again and again, I'm trying to understand
that, this is very important, not to be very enthusiastic. But concentrate on whatever
that was not able to achieve. - Patient.
- Yeah. And understand why. And now I understand that, because I believe in math, I believe that, in this idea. But now when I see that there are only two ways of convergence,
and we're using loss. That means that we must
as well as people do it. But now, exactly in
philosophy and what we know about predicate, how we
understand life can be described as a predicate. I thought about that. And that is more or less
obvious level of symmetry. But next, I have a feeling it's
something about structures. But I don't know how to formulate, how to measure and measure a
structure and all this stuff. The guy who will solve this challenge problem, then when they will look at how he did it, probably just only symmetry is not enough. - [Lex] But something like
symmetry will be there. Structures of that kind.
- Oh yeah, absolutely. Symmetry will be there. A level of symmetry will be there. And level of symmetry, anti-symmetry, diagonal, vertical, I even don't know how you can use in different direction the degree of symmetry;
that's very general. But it will be there. I think that people are very sensitive to the idea of symmetry. But there are several ideas like symmetry. As I would like to learn. But you cannot learn
just thinking about that. You should do challenging problems and then analyze them. Why it was able to solve them. And then you will see. Very simple things, it's not easy to find. (Lex laughs)
Even talking about this, every time.
- Yes. - I was surprised, I tried to understand, these people describe in language strong convergence mechanism for learning. I did not see it, I don't know. But weak convergence, the duck story and story like that when you will explain, you will use weak convergence argument. It looks like a (mumbles) but when you try to formalize, you're just ignoring this. Why? Why 50 years? From start of machine learning. - [Lex] And that's the role of philosophy, thinking about life.
- I think that might be. I don't know. Maybe this is theory also, we should blame for that because
empirical risk minimization and now just starting, if
you read now textbooks, they just about bound about
empirical risk minimization. They don't look for another
problem like admissible set. - But on the topic of life, perhaps we, you, could talk
in Russian for a little bit. What's your favorite
memory from childhood? Okay, I want you to be
my (speaks in Russian) - Music. - How about, can you try
and answer in Russian? (speaking in Russian) What kind of musica? (speaking in Russian) (speaking in Russian) Now that we're talking about Bach, let's switch back to English, 'cause I like Beethoven and Chopin so. - [Vladimir] Chopin is
another amusing story. I was--
- But Bach, if we talk about predicates, Bach probably has the most sort of, well defined
predicates that underlie it. - You know, it is very interesting to read what critics writing about Bach, which words they're using, they're trying to describe predicates.
- Yeah. - And then Chopin. It is very different vocabulary. Very different predicate. And I think that, if you will make collection of that. So maybe from this you can describe predicate for digit recognition. - [Lex] From Bach and Chopin. - No, no, no, not from Bach and Chopin. - [Lex] From the the critic interpretation of the music, yeah.
- They're trying to explain you music, what they use this? They describe high-level
ideas of Plato's ideas behind this music.
- That's brilliant. Art is not self-explanatory in some sense. So you have to try to
convert it into ideas. - It is insulate problems. When you go from ideas to, to the representation. It is easy way, but when
you're trying to go back, it is you will pose problems but, nevertheless, I believe when
you're looking from that, even from art, you will be able to find predicate for digit recognition. - That's such a fascinating
and powerful notion. Do you ponder your own mortality? Do you think about it? Do you fear it? Do you draw insight from it? - About mortality? Oh yeah. - [Lex] Are you afraid of death? - Not too much, not too much. It is pity that I will not be able to do something which I think I was a feeling to do that. For example, I will be very happy to work with guys, take tradition from music. To write this collection of description, how they describe music,
how they use a predicate. And from art as well. Then take what's in common, and try to understand predicate which is absolute for everything. - [Lex] For visual recognition and see that there is a connection. - Yeah, yeah, exactly. - [Lex] There's still
time; we've got time. (laughing) We've got time. - It takes years and years and years.
- You think so? - It's a long way. - See, you've got the
patient mathematician's mind. I think it could be done very quickly and very beautifully. I think it's a really elegant idea. Some of many.
- Yeah, you know, the most time it is not
to make this collection, to understand what is in common, to think about that once again and again and again.
- Again and again and again. But I think sometimes,
especially when you just say this idea now, even just
putting together the collection and looking at the different sets of data. Language, trying to interpret
music, criticize music, and images, I think there will be sparks of ideas that will come. Of course, again and again you'll come up with better ideas but
even just that notion is a beautiful notion.
- I even have some example. So I have friend, who was specialized in Russian poetry. She is professor of Russian poetry. She did not write poems, but she know a lot of stuff. She make books, several books and one of them is a collection of Russian poetry. She has images of Russian poetry. She collected all images
of Russian poetry. And I asked her to do following. You have Nip's digit recognition. And we get 100 digits, less than 100, I don't
remember, maybe 50 digits. And try from practical point of view, describe every image you see, using only words of
images of Russian poetry. And she did it. And then we tried to, I call it learning using
privileged information. I call it privileged information. You have on two languages. One language is just image of digit. And another language by it
a description of this image. And this is privileged information. And there is an algorithm
when we are working with privileged information,
you're doing well. Better, much better. - So there's something there.
- Something there. And there is and the thing, she unfortunately died. The collection of digits and poetic descriptions of those digits. - [Lex] So there's something there in that poetic description. - I think that there is an abstract ideas on the Plato level of ideas.
- Yeah, that are there, that could be discovered, and music seems to be a good entry point. - But as soon as we start this, here's this challenge problem. - The challenge problem--
- It immediately connected to all this stuff. - Especially with your
talk and this podcast, I'll do whatever I can to advertise. Such a clean, beautiful,
Einstein-like formulation of the challenge before us.
- Right. - Let me ask another absurd question. We talked about mortality,
we talked about philosophy of life; what do you think
is the meaning of life? What's the predicate for mysterious existence here on Earth? - I don't know. It's very interesting how we have, in Russia, I don't know if you know the guy Strogatski? They're writing futures
they're thinking about, Hume, what's going on. And they have an idea that there are developing two type of people: Common
people and very smart people. They just started. And these two branches of people will go in different directions very soon. So that's what they're
thinking about next. - (laughs) So the purpose of life is to create two (chuckles) two paths. - Two paths.
- As human societies. Yeah.
- Yes, simple people and more complicated people. - Which do you like best? The simple people or the complicated ones? - I don't know, Strogatski, he's just, he's fantasy but you know, every week we have guy who is just writer, and also Soletskoff literature. And he explained how he
understands literature and human relationship. How he sees life. And I understood that I'm just
small kid comparing to him. He is very smart guy
in understanding life. He knows this predicate, he
knows big blocks of life. I'm amused every time I listen to him. And he's just talking about literature. And I think that I was surprised. So the managers in big companies, most of them are guys who study English language and English literature. So why? Because they understand life. They understand models. And among them, maybe
many talented creatures, which are just analyzing this. And this is big science like Propp did. This is his blocks. Yes, very smart. - It amazes me that you are and continue to be humbled by the brilliance of others. - I'm very modest about myself. I see so smart guys around. - Well let me immodest for you. You're one of the greatest
mathematicians/statisticians of our time, it's truly an honor. Thank you for talking.
- No, no, no, okay, okay. - And let's talk.
- (laughs) It is not. - Yeah, let's talk.
- I know my limits. - [Lex] Let's talk again when
your challenge is taken on and solved by a grad student. - Let's talk again.
- Especially when-- - [Vladimir] I hope that this happens. - Maybe music will be involved. Vladimir, thank you so much. It's been an honor.
- Thank you very much. - Thanks for listening
to this conversation with Vladimir Vapnik. And thank you to our
presenting sponsor Cash App. Download it, use code: LexPodcast. You'll get $10 and $10 will go to FIRST, an organization that inspires and educates young mind to become
science and technology innovators of tomorrow. If you enjoyed this podcast,
subscribe on YouTube, give it five stars on Apple
PodCast, support it on Patreon, or simply connect with me
on Twitter @LexFridman. And now let me leave you with some words from Vladimir Vapnik. "When solving a problem of interest, "do not solve a more general problem "as an intermediate step." Thank you for listening. I hope to see you next time.