What are "volatile" learning curves indicative of? I have a dataset set with ~40 features onto which I'm applying a multi-layer perceptron for regression purposes. The train, validation, and test sets are made up of 3M, 800K, and 800K examples each, respectively. At the end of 50 epochs, the loss for training, validation, and testing is at 0.1436, 0.1433, and 0.1422 respectively---which makes me think that the model generalizes quite well out of sample.
This said, the learning curves look very volatile, and the downward loss over the epochs is barely visible even with when smoothing the plots. What is this indicative of? According to this blog, it could be because the validation set is too small. However, as mentioned earlier, I already have 800K examples in the validation (i.e., ~1/4 of the training set's size). Also, the validation learning curve does go down, albeit in a very "jagged" way, unlike what is shown on the blog.
Here are the learning curves in question:


This means that your optimization hasn't really settled, yet. There could be many causes for that. If you are sure that the validation set is from the same population as the training set, then the problem lies in the model and optimization. Measures I would consider are:

*

*Just be patient, give it more time, maybe it is sorted out with time.

*Add more regularization.

*Maybe your step size is too large. Reduce your learning rate.

*Ordinary MLPs often don't settle anymore if they contain too many layers. Consider switching to a better architecture.

*Google one of the DLL tips and tricks documents, and go through all the measures that are suggested there, checking whether they improve the convergence.

