Unfortunately, GPUs are very good at doing large simple operations like element-wise ops or mm but very bad at smart things like indexing.
For the time noise, I don’t see any way around this, as I said, this is most likely some hardware quirks and I don’t know GPU internals enough to have an idea why this happens