It depends on your model implementations and the used operations.
To isolate it further you could use nvtx markers and check where the synchronization is happening (via torch.cuda.nvtx.range_push and torch.cuda.nvtx.range_pop).