The spikes in the training set went way after reprocessing the dataset and training the model off of the newly processed training set, but the spikes did not go away after shuffling the data.
Then it seems that the unbalanced dataset still preforms less than the balanced dataset in terms of loss.
What can I do to get my accuracy above 80%?