Resurgence of Neural Networks
Really interesting stuff.I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). This RBM approach sounds better than back-propagation, but it also sounds like it would be prone to runaway feedback.One of the performance problems with neural networks is that the number of cores on a typical machine are far less than the number of input and intermediate nodes in the network. The output nodes are less of a concern as you're trying to distill a lot of data down to a little data, but there is no reason to treat them differently. There are (very few) examples of NNs on GPUs, so that helps, but I've recently been curious to try a different, more hardware-driven approach, just because one could.Texas Instruments has a cheap DSP chip that you'ns are probably familiar with called the MSP430. It's pretty easy to use, the tool chain is free and fairly easy to setup (especially for a bunch of professional software devs like us, right? right? Well, there's an Arduino-like tool now, too, if not), costs around 10 cents in bulk for the simplest version, requires very few external parts to run (something like a power source, 1 cap, and two resistors), and it has a couple of serial communication protocols built in. I'm quite fond of the chip; I've used it to build a number of digital synthesizers and toys.For about $50 and quite a bit of soldering time, you could build a grid of 100 of these, each running at 16Mhz, and I bet with a clever design you could make them self programmable, i.e. propagate the program EEPROM over the grid. Load up a simple neural network program, maybe even having each chip simulating more than one node, and interface it with a PC to pump data in one end and draw it out the other. It might not be more useful than the GPGPU approach, but having physical hardware to play with and visualize node activity through other hardware would be a lot of fun.