The thing I always look to is arguably the most popular PyTorch extension - torchvision 1. Building it is much easier than building PyTorch and you’ll be able to observe what it is doing.
In fact, I would recommend to imitate TorchVision’s method of building the extension.
Best regards
Thomas