thank you @lewtun. I have gone through their implementation and noticed that their per_device_train_batch_size is 1. Once I changed that, It works and I am actually getting some amazing results (its on its final epoch now, but currently hitting 80% accuracy   ).
I was hoping you could perhaps explain to me what this per_device_train_batch_size does and why was that the issue?