Creating an MRE! indeed do wonders!
OK, it turns out there was no problem to begin with! the error was comming from the other section of the script that expected a cuda while I was giving it a model in cpu mode and thus the error.
silly me! completely missed that!
Thanks a lot for your generous  help @ptrblck really apprecaite it.