Proving the Lottery Ticket Hypothesis: Pruning is All You Need
I&#x27;ve always felt the there is a deep connection between evolution and thought, or more specifically, genetic algorithms (GAs) and neural networks (NNs).<p>The state of the art when I started following AI in the late 90s was random weights and hyper-parameters chosen with a GA, then optimized with NN hill climbing to find the local maximum. Looks like the research has continued:<p><a href="https:&#x2F;&#x2F;www.google.com&#x2F;search?q=genetic+algorithm+neural+network" rel="nofollow">https:&#x2F;&#x2F;www.google.com&#x2F;search?q=genetic+algorithm+neural+net...</a><p>All I&#x27;m saying is that since we&#x27;re no longer compute-bound, I&#x27;d like to see more big-picture thinking. We&#x27;re so obsessed with getting 99% accuracy on some pattern-matching test that we completely miss other options, like in this case that effective subnetworks can evolve within a larger system of networks.<p>I&#x27;d like to see a mathematical proof showing that these and all other approaches to AI like simulated annealing are (or can be made) equivalent. Sort of like a Church–Turing thesis for machine learning:<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Church–Turing_thesis" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Church–Turing_thesis</a><p>If we had this, then we could use higher-level abstractions and substitute simpler algorithms (like GAs) for the harder ones (like NNs) and not get so lost in the minutia and terminology. Once we had working solutions, we could analyze them and work backwards to covert them to their optimized&#x2F;complex NN equivalents.<p>An analogy for this would be solving problems in our heads with simpler&#x2F;abstract methods like spreadsheets, functional programming and higher-order functions. Then translating those solutions to whatever limited&#x2F;verbose imperative languages we have to use for our jobs.<p>Edit: I should have said &quot;NN gradient descent to find the local minimum&quot; but hopefully my point still stands.<p>Edit 2: I should clarify that in layman&#x27;s terms, Church-Turing says &quot;every effectively calculable function is a computable function&quot; so functional programming and imperative programming can solve the same problems, be used interchangeably and even be converted from one form to the other.