If you would like to keep only e.g. the first 4 classes, you could manipulate the last conv layer:
with torch.no_grad():
    fcn.classifier[4].weight = nn.Parameter(fcn.classifier[4].weight[:4])
    fcn.classifier[4].bias = nn.Parameter(fcn.classifier[4].bias[:4])