Downloading the source and building it after your manipulation might work, but the easier way would probably be to create a custom nn.Module and use the torchvision model as the base class.
Inside your custom model you could then redefine the internal parameters and methods.