Update: Increasing the batch size to 256, thanks to gradient-accumulation, improved the performance 
"--per_device_train_batch_size=16 "
"--per_device_eval_batch_size=16 "
"--gradient_accumulation_steps=16 "