The pytorch/builder 1 repository contains the build scripts for the pip wheels and conda binaries, so you could compare your setup to the official one and also use the docker container workflow for the build.