Question: I have to train a very large feedforward neural network for pattern recognition.

The network has 10,000 input neurons, 200 output neurons, the hidden layer of it has 5100 neurons. Function x / (1 + abs(x)) is used as the activation function.

The size of the training data set is 100,000.

The problem is it took very long time to train the neural network, which makes fine tuning the network very time consuming.

Two possible options came to me for improving training performance:

* Reduce the size of hidden layer to somewhere between 500 and 1000. This would dramatically reduce the number of multiplication operations needed. However I'm not sure whether such reduction would harm recognition accuracies.
* Get a high performance matrix multiplication library (maybe even introduce OpenCL/CUDA), currently only naive matrix multiplication is implemented, which should be slower than a carefully tuned one.

Are there any better options ? I'm new to ANN, any suggestions are appreciated.

Thanks.

Answer: 
In one sentence - **first play with the size of the hidden layer to see what size fit your needs**.

I've once written an implementation to the feed-forward back-propagation. Size of the network may be the only factor, but you also have etta (factor of convergence) in the back-propagation phase, and other factors (which I need to remember, but I promise to try). Also you have some tricks as cross-correlations. Also you can try to optimize the differentiation code and the activation function (small x may introduce numeric issues, and you have division operations)

My strategy would be first examine the hidden layer - 5000 neurons in the hidden layer compared to 10000 input neurons is quite a massive hidden layer (however, it depends on the problem).
The number of functions to compute is the major bottleneck - quadratic to the size of the hidden layer. if number of nodes in the hidden layer is O(N), then the number of functions to to perform between the outer layers and the hidden layer is O(N^2).
This is because each node in the input layer needs to contribute to all of the nodes in the hidden layers, and the same with the output layer and the hidden layer.

In the problems I dealt with - image compression, 1/8th or even 1/16th of the size of input neurons satisfied my needs. I highly recommend you to first test the hidden layer size to see where you get the optimal tradeoff between hidden layer size and result quality.

Also - **feed forward (and back propagation) can be implemented in a parallel manner** - consider using open-mp for that.
The accumulation of the weights in each node can be done very efficiently with parallel processing.
Take a loot at <http://deeplearning.net/software/theano/>. 

It does support OpenCL and is tailored quite well for ANN, e.g. here is the LeNet implementation:
<http://deeplearning.net/tutorial/lenet.html>