  This paper describes the design and implementation of parallel neural
networks (PNNs) with the novel programming language Golang. We follow in our
approach the classical Single-Program Multiple-Data (SPMD) model where a PNN is
composed of several sequential neural networks, which are trained with a
proportional share of the training dataset. We used for this purpose the MNIST
dataset, which contains binary images of handwritten digits. Our analysis
focusses on different activation functions and optimizations in the form of
stochastic gradients and initialization of weights and biases. We conduct a
thorough performance analysis, where network configurations and different
performance factors are analyzed and interpreted. Golang and its inherent
parallelization support proved very well for parallel neural network simulation
by considerable decreased processing times compared to sequential variants.

Neural and Evolutionary Computing;Distributed, Parallel, and Cluster Computing