It seems that doing pip uninstall torch still left some old files behind that were conflicting with the new installation. Removed the conda environment and created a new one from scratch and now pytorch works nicely.