Sorry for not answering earlier, but the problem “randomly” got solved after uninstalling and re-installing pytorch.
Now I am having again this same issue, after a few non-obvious changes to my code.
I have tried building a minimum version with no success so far, GPU works fine.