I don’t know if in earlier versions of PyTorch the following works, but in v1.6 deleting a layer is as simple as:
del net.avgpool

This both removes the layer from model.modules and model.state_dict.
This is also does not create zombie layers, as an Identity layer would do; Simplifying model loading.