Thanks for the detailed analysis.
I had a quick look at the implementations and besides the missing synchronizations, I’m still trying to figure out, if the CUPY arrays can hold any CPU data, or if it’s restricted to the device only?
E.g. while this line of code
gadgetCPU.insertData(u_embeddings.weight.data,  indexess )

seems to copy the CUDATensor from u_embeddings onto the CPU, it seems as if the underlying data never left the GPU:
gadgetCPU.CUPYcorpus.device
Out[4]: <CUDA Device 0>

I’m not really experienced in CUPY so I might miss something.