The Principles of Deep Learning Theory
The level of analysis deep learning systems get is considerably out-of-proportion, and speaks to a superstitious view of how ML works. Namely, that it uncovers latent representational structure not present in the data.<p>(Incidentally, its clear this book doesnt make this mistake, already identifying that NN is basically kNN).<p>Rather ML algorithms are just rememberings of data coupled with various degrees of compression, with kNN having zero compression -- just a straight weights=data; and NNs having fairly significant levels, where weights=ensemble(compressions(data)).<p>We should therefore regard the ML step as incredibly trivial, it is just a clever process of averaging over the data given to it. The whole &quot;magic&quot; of ML, such as it is, is *only* in the data. And this is where the word &quot;data&quot; hamstrings our ability to see properly.<p>Everything *isnt* data; and &quot;data&quot; isnt some source of information. The world exists, and &quot;data&quot; is just what we call <i>any</i> measurement of <i>any</i> part of it by <i>any</i> means. &quot;Data&quot; is only *relevant* to the probelm we&#x27;re trying to solve if we do incredible amounts of experimental work to carve-the-world along its joints, ie., to have the right concepts; and incredible amounts of work to measure along its joints, ie., to have the right units. *And then* to eliminate all the coincidences and irrelevances. *And then* to provide that to a machine, which at this point, does basically nothing but automate our effort.<p>Almost all data it is possible to collect is useless, indeed, an infinite amounts is useless. The magic of ML is a sleight-of-hand trick -- we dont really need to know how its averaging of our data does anything useful -- it almost never does.<p>Rather, it is our &quot;experimental design&quot; which produces the usefulness of the system. ML algorithms are just interpolations and averages through data <i>prepared</i> to produce useful averages by (literally millenia) of human ingenuity.<p>It takes actual intelligence to do this because the world isnt data, and almost any measurement one cares to make (with eyes, even) produces endless ambiguities and coincidences that you have to &quot;be in the world&quot; to resolve; and resolution is a dynamic process which you &quot;have to be here for&quot;.