Balanced datasets have not solved my issue.
This is one epoch of validation, and it looks just like the batches from before.

stillNotWorking.png363×655 27.8 KB

What could be the issue ?
For comparison, this is a graph using Transfer Learning from the Places365 dataset:

transfer.png374×659 32.3 KB

This has 8 epochs of training and validation.