Yes, essentially you are building a very large computational graph until backward is called. I don’t know what the value of K is, but it’s possible the the GPU version is dominated by data transfers from host to device.