Neural network gives very different accuracies if repeated on same data, why? I'm running a neural network to classify audio files into 4 classes. This uses 3300 1min files split roughly evenly across classes. I split this into 80:10:10 train:validation:test sets. This trains for 50 epochs, I save the model from the epoch with the best validation accuracy, then inference on the test data using this model.
However if I run repeats of the network, the accuracy is often wildly different. It may be 98% on one repeat, then 16% on the next, averaging out at about 57% across several repeats. This occurs for train, val and test data. Its as if it often fails to train, but sometimes has a lucky breakthrough.
Using some predefined features and random forests I can get about 83% accuracy with 0.05 StDev so I know the task can be completed consistently to a good accuracy. When I run the neural net to classify just two classes it also consistently performs well (89%).
The network I am using is adapted from this github repo (with added bells and whistles to get accuracy etc), which itself is adapted from the VGG image classifier. I have bug tested everything I can think of, and if I run it using repeats of the same audio (e.g 3 files from each class repeated 100 times) and put these in the val/test data as well it does a great job (99% acc).
Any ideas on what could be causing these huge differences in accuracy and how to fix it? I'm somewhat new to deep learning so may have overlooked simple ideas.

It means that the neural network has a high variance. It is prone to overfitting, it sometimes picks the right patterns, sometimes the noise. It can be due to using different training data, random initialization, or both. That is why procedures, as employed by yourself, are useful, if you trained and validated it only once you would never know. What this means for you is that you don't really know how the model would behave at prediction time.
You may check the What should I do when my neural network doesn't generalize well? thread for possible solutions.
