A 'Brief' History of Neural Nets and Deep Learning
Excellent article, thanks for the contribution.<p>Perhaps some independent validation, but I was coincidentally having this conversation the other day with a relatively well known computer vision researcher, about why it seems like the idea of neural nets has floundered for decades and suddenly it's the hot topic, and we're seeing massively improved results.<p>His answers, summarized, are that:<p>1- Big data is making possible the kind of training we could never do before.<p>2- Having big data & big compute has made some training breakthroughs that allowed the depth to increase dramatically. The number of layers was implicitly limited until recently because anything deep couldn't be practically trained.<p>3- The activation function has very commonly in the past been an S-curve, and some of the newer better results are using a linear function that is clamped on the low end at zero, but not clamped on top.<p>All really interesting to me. This is making me want to implement and play with neural nets!<p>Of course, now the big question: if we have a neural net big enough, and it works, can we simulate a human brain? (Apparently, according to my AI researcher friend, we're not there yet with the foundational building blocks. He mentioned researchers have tried simulating a life-form known to have a small number of neurons, like a thousand, and they can't get it to work yet.)