I don’t know why it shouldn’t work, as I’m using a TitanV myself with the binaries, different CUDA versions, and builds from source.
If CUDA_VISIBLE_DEVICES is not working, you could try to disable or remove the old GPU for the sake of debugging and rerun your script.