So looks like it was a problem with tuning. Turns out I had to tune hyperparameters even more. A Batch size of 16 coupled with a learning rate of 0.001 seems like its working. Is a more structured approach to do a hyperparam search in something like tensorboard, or does anyone have better suggestions when hitting road blocks like this?