Suppose I have a Modules class as below:
class Bottleneck(nn.Module):
    def __init__(self, nChannels, growthRate):
        super(Bottleneck, self).__init__()
        interChannels = 4 * growthRate
        self.bn1 = nn.BatchNorm2d(nChannels)
        self.conv1 = nn.Conv2d(nChannels, interChannels, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(interChannels)
        self.conv2 = nn.Conv2d(
                interChannels, growthRate, 3, padding=1, bias=False)

    def forward(self, x):
        out = self.conv1(F.relu(self.bn1(x)))
        out = self.conv2(F.relu(self.bn2(out)))
        out = torch.cat((x, out), 1)
        return out

Then in IPython REPL:
In [6]: a = Bottleneck(1, 2)

In [7]: for b in a.modules():
    print(b)
   ...:     
Bottleneck (
  (bn1): BatchNorm2d(1, eps=1e-05, momentum=0.1, affine=True)
  (conv1): Conv2d(1, 8, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (bn2): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True)
  (conv2): Conv2d(8, 2, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
)
BatchNorm2d(1, eps=1e-05, momentum=0.1, affine=True)
Conv2d(1, 8, kernel_size=(1, 1), stride=(1, 1), bias=False)
BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True)
Conv2d(8, 2, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)

I read the source code of the Module class, I do not find any code show that how the Module subclass add these Modlue member like “BatchNorm2d” and so on to its _modules.