On Chomsky and the Two Cultures of Statistical Learning (2011)
I think that Norvig hits the nail on the head near the beginning of his piece:<p>&quot;I believe that Chomsky has no objection to this kind of statistical model [the Newtonian model of gravitational attraction]. Rather, he seems to reserve his criticism for statistical models like Shannon&#x27;s that have quadrillions of parameters, not just one or two.&quot;<p>This is no more than an objection to problems of fitting your chosen model to data. If you only have a small number of free parameters, then you can fit your model with a reasonable amount of data. If you have a large number of parameters then you have to introduce some extra assumptions, as Norvig (of course) acknowledges slightly earlier (described as &quot;smoothing&quot;, in context):<p>&quot;For example, a decade before Chomsky, Claude Shannon proposed probabilistic models of communication based on Markov chains of words. If you have a vocabulary of 100,000 words and a second-order Markov model in which the probability of a word depends on the previous two words, then you need a quadrillion (10^15) probability values to specify the model. The only feasible way to learn these 10^15 values is to gather statistics from data and introduce some smoothing method for the many cases where there is no data.&quot;<p>Thus, although both models are statistical, it is much easier to have confidence in Newton&#x27;s law of gravitation than it is in a Markov model of some communication channel, because the data tell a clear picture. The imprecision of Newton&#x27;s law in certain parts of the problem space (unobserved during his time) is a moot point - any such objections apply equally well to models with many parameters, and then you _still_ have to accept that you have made extra assumptions &quot;outside&quot; the scope of your model.<p>If you can explore your entire problem space, then you can build a complete &quot;model&quot;. If not, then having more parameters than data _requires_ additional assumptions. Chomsky&#x27;s point stands.