Your problem is probably elementwise assignments to cuda tensor, like you do with dense weights above. You should do these on cpu tensor, for one-hot you can even use bool dtype, then do .to(device=“cuda”, dtype=XXX). Actually, I don’t know if such sparse/dense matmul is faster on cuda, especially with float64.