A 'Brief' History of Neural Nets and Deep Learning
Excellent article, thanks for the contribution.<p>Perhaps some independent validation, but I was coincidentally having this conversation the other day with a relatively well known computer vision researcher, about why it seems like the idea of neural nets has floundered for decades and suddenly it&#x27;s the hot topic, and we&#x27;re seeing massively improved results.<p>His answers, summarized, are that:<p>1- Big data is making possible the kind of training we could never do before.<p>2- Having big data &amp; big compute has made some training breakthroughs that allowed the depth to increase dramatically. The number of layers was implicitly limited until recently because anything deep couldn&#x27;t be practically trained.<p>3- The activation function has very commonly in the past been an S-curve, and some of the newer better results are using a linear function that is clamped on the low end at zero, but not clamped on top.<p>All really interesting to me. This is making me want to implement and play with neural nets!<p>Of course, now the big question: if we have a neural net big enough, and it works, can we simulate a human brain? (Apparently, according to my AI researcher friend, we&#x27;re not there yet with the foundational building blocks. He mentioned researchers have tried simulating a life-form known to have a small number of neurons, like a thousand, and they can&#x27;t get it to work yet.)