What can I do when Overfitting doesn't seem to go away by any means? So first of all I've seen a lot of overfitting questions around here, but none of the answers seem to improve my model.
I wrote a neural network made without frameworks (only used numpy), and for the past two or three weeks I've been trying to train it to recognize digits and letters from a license plate dataset. The set is pretty small (~800 images), but a model with only one layer was enough to give a pretty high accuracy.
The problem is when I use it to predict on the cross validation set, as both the cost function and accuracy stay pretty much constant all the way.
The cross validation set / dev set was created by following the next steps:

*

*Load whole dataset,

*Shuffle samples,

*Split the dataset (70% training set, 30% dev set).
This way I avoid having a dev set with a different distribution, as they both come from the same dataset.

This is the training set cost vs dev set cost plot, with the dataset normalized between 0 and 1 (dividing by 255):

Now this is the same plot if I normalize the images using the formula pixel = (pixel-mean)/sqrt(variance)


For the next plots I used the first normalization technique, as it gave better results.
To solve overfitting, I tried:

*

*L2 Regularization: doesn't do much except for increasing training time, I adjusted the learning rate so you can see the overfitting.



*

*Dropout: I was betting on this one, as I've read it's good for computer vision and also for small datasets, but in practice training just takes a lot longer, and I don't have many parameters to try as my network is too shallow. The following plot is by training with dropout of 20% of the input layer each time forward prop is applied.



*Batch Norm: Here I'm normalizing the Z vectors inside the model, by calculating the mean and the variance and replacing each Z for (Z-mean)/sqrt(variance).


*

*Early stopping: doesn't work as the dev cost goes down but only like 1% or so.


*Simplifying the model: I just can't do this one, as it's as simple as it can be at the moment, the topology is [input_layer, output_layer]. The number of features is 2048 at the moment (64 by 32 px images), and the output layer has 36 units (one per each letter + the 10 numbers). I tried scaling the images down to get less features (down to 16 by 8) but the results were pretty much the same.


*Deeper network + dropout: And one last option, I added two more layers to the model (now the topology is [input_layer, 40, 40, output_layer]. The dropout percentage for each layer is [20%, 40%, 40%, 0%].

No matter what I try, it always either improves or damages learning time on the training set, but the dev set remains pretty much unaltered. I forgot to mention, as the dataset is small and I'm using vectorization, I'm using batch gradient descent, as I see no advantages to using mini batches in this case.
What am I doing wrong? Is my dataset flawed?
EDIT:
So I tried the same stuff with a more standard dataset, MNIST, and I'm getting the same results. Could it be that my neural network library is broken or has a bug? I don't really know how to check because it is learning and predicting correctly, but only on the training set, so backprop and gradient descent is working as intended...

I finally solved the issue a few months ago, I forgot to post here. The problem was I was merging part of the input data together by misusing the reshape method of Numpy. So in short the model was learning from basically random data, and it was impossible for it to perform well in the validation set under any circumstance.
