What I learned with some experience of mine are (including above advice)

Try to map each operation to matrix multiplication as much as possible
avoid gather and select
don’t let GPU wait for CPU batch preparation, make sure that batch data is processed beforehand or done in background process. Keep GPU busy all the time. “nvidia-smi” is your friend to check GPU usage.