My current solution is to prepare the pruned module for a successful deepcopy by deleting the non-leaf tensor as an attribute of the module (since it is neither a parameter nor a buffer of the module)
import copy
import torch
import torch.nn.utils.prune as prune

foo = torch.nn.Conv2d(2, 4, 1)
foo2 = copy.deepcopy(foo) # copy is successful before pruning

pruner = prune.L1Unstructured.apply(foo, name='weight', amount=0.2)
foo3 = copy.deepcopy(foo) # copy fails after pruning

print(pruner._tensor_name) # 'weight'
delattr(foo, pruner._tensor_name)
foo4 = copy.deepcopy(foo) # copy is successful

print(hasattr(foo4, pruner._tensor_name)) # False
# Run forward() once to recover the deleted attribute 
_ = foo4(torch.empty(1, 2, 4, 4))
print(hasattr(foo4, pruner._tensor_name)) # True

Note that the deleted attribute is reset during the first forward call on the module. This is because it is being reregistered at every call in the forward_pre_hook that the BasePruningMethod class implements here 1
cc: @Michela, any thoughts would be much appreciated for a longer term solution