If you need to build PyTorch from source, I would remove it from the environment file, create the conda environment, activate it and finally build PyTorch in from source using the activated environment.
However, I’m wondering, if you really need to build PyTorch from source for the fastai lectures.
What is your use case for this?