You could copy the torchvision VGG implementation from here 2 and adapt it to your needs (e.g. by removing/replacing layers etc.).
Another approach would be to create the torchvision model object and manipulate the layers directly, but I think this approach would be more suitable for smaller changes, such as replacing the last linear layer for transfer learning.