By “finetune the model” I assume you are reinitializing the last layer?
Also, let’s assume you are freezing all preceding layers and just train the last new classification layer (nn.Conv2d with out_channels=2 in your case).
The “feature extractor” part of the model, so basically all layers before the final classifiers, are already trained on the COCO dataset and can successfully yield useful features for the classification of the dataset.
If you swap the last classification layer for a new one adapted to your use case, these incoming features will be used to train the classification layer so that is minimizes the loss for your segmentation use case.
The last layer won’t have any knowledge about the trained classes in the past, but the other layers might yield good features for your current classes.
Side note, unrelated to your original question:
Note that this approach might still work, if you change the data domain, e.g. if you would like to segment organs in a CT scan now.
In such a case, I would recommend to try to finetune the complete model, if you have enough data, as the features from the first layers might not be suitable for the CT images.