the per_device_train_batch_size specifies the batch size for the device you are training on (e.g. GPU/TPU/CPU), so if your training set has 1,000 examples and and per_device_train_batch_size=1 then it will take 1,000 steps to complete one epoch.
by increasing the value of per_device_train_batch_size you are able to train faster since it takes less steps to complete each epoch (e.g. if per_device_train_batch_size=4 then we only need 250 steps / epoch in our example), but this can sometimes lead to worse performance since the gradients are averaged / summed in a minibatch.
in your case, my guess is that with per_device_train_batch_size=1 you need to train for a very long time to see the model learn anything.