You two hypothesis are very good candidates. Keep in mind that GPU are really good at doing a single large op but very bad at doing many small ones.  So removing the for-loop will definitely help.
Moving Tensors between cpu and gpu is quite expensive as well and should be avoided yes.