Generally, you might not see an expected speedup on the GPU, if you are using a lot of “small” operations and are thus limited by the memory bandwidth.
That being said, even with a batch size of 1 you would see a speedup, if the computational workload of the model is not tiny.
Could you post your model definition so that we could have a look at it, please?