Some operations like .item() will add a synchronization point in your code.
Could you post the model definition, so that we could have a look for unwanted sync points?
Also, how large is your workload?