Yeah, I wouldn’t expect to save so much time by slicing the last conv layer.
Did you time it?
If so, what was the difference?
To use specific class indices, just pass them as a list:
with torch.no_grad():
    fcn.classifier[4].weight = nn.Parameter(fcn.classifier[4].weight[[0, 3, 16]])
    fcn.classifier[4].bias = nn.Parameter(fcn.classifier[4].bias[[0, 3, 16]])