One last thing as well is that the blas libraries (like mkl) have a tendency to aggressively use compute cores and I think that in a multithreaded environment, they can actually slow each other down.
You definitely want to do batching if possible to avoid this.
Will keep you updated if I get further ideas.