First step forward. It turns out that doing .to(device) to all data first to GPU instead of inside the forward pass reduces GPU time to 2.74 from  38.33. Still, the CPU is slightly faster. Next try is vectorization to avoid the for loop and I will report back.