Hi,
In Net2 and Net3, they are not registered to optimizer. They are not Parameters at all. Only when they are registered to Parameters before calling optim.SGD(model.parameters(), lr = 0.01, momentum=0.9), then it get updated when running optimizer.step().
Even for MyNet4, conv2_filters are not updated as usually we define the parameters to be updated right after the definition of the network.
class MyNet4(nn.Module):
    def __init__(self):
        super(MyNet3, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.relu1 = nn.ReLU(True)
     #  move this to forward
     #  self.conv2 = nn.Conv2d(64, 1, 3, padding=1)

    def forward(self, input):
        # I change its type to Parameter
        self.conv2_filters = nn.Parameter(torch.randn(1, 64, 3, 3))
        return F.conv2d(self.relu1(self.conv1)), self.conv2_filters, padding=1)