That is the fresh thought, I didn’t aware of the existence of nn.Identity. 
Maybe the code will be like this… (This is for my future. I usually revisit my thread to copy some snippet)
class Model(nn.Module);
  def __init__(self):
    super().__init__()
    self.w = nn.Linear(10, 5)
    self.tracker_act = nn.Identity("exp_after_linear")

  def forward(self, x):
    x = self.w(x)
    x = self.tracker_act(torch.exp(x))
    return x

act = {}
def collect(m, i, o):
  return act["exp_after_linear"] = o.detach().cpu()

model = Model()
model.tracker_act.register_forward_hook(collect)

I think it is nice. 
Thanks for the nice discussion, @albanD and @ptrblck.