Thanks for your reply. But  what I want to is :
 class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        if self.training:
            self.auxiliary = nn.Conv2d(in, out)

The auxiliary layer is only defined for training state.