First, if the loading data in the memory or other preprocessing steps on the CPU are the bottleneck, then it can reduce the GPU usage. Since in this case, the GPU has to wait for the data to be pre-processed and sent over.
Second thing, how much of the GPU memory is utilised? If there is still some part of GPU memory that is still available, you can increase the batch-size, so that the GPU will do more work per batch.