GPU is optimized for large matrix multiplication, so it might be not too surprising if it’s slower in other scenarios? 
PS: that’s why loop hits like crazy, that might be why, can you batch it or something