Solved by ptrblck in post #6 
              
              
                Yes, that;s correct. While this might work for resnet50, it might not for other architectures. 

That might be the simplest method. Classification models usually have a classification layer created as model.fc or model.classifier, which you could replace with your custom convs.