Thanks for reply! I use torch.detach() to cut the graph every iteration, and I think that do the same thing as your suggestion. My implementation of the RNN part wasn’t practical, I’ll unrolling it to finite steps like RCNN later.
However, about question 1. I find that the gradient can actually go backward. if I add a fc layer after the gap layer, then the gradient looks normal, and the loss can be reduced. But if not, the out put scores of gap layer become all zeros, thus all the weights.grad in the network before gap layer become zero very soon(about 3 or 4 iteration, batch size 64), and the loss stuck in one value(0.6931, every time I try, always this exactly one value, loss begin at about 0.70). I know this is not a question about PyTorch, but I’m really lost on this, why the features become zero? Why adding a fc layer can solve this problem?
Here is the simplified code, I remove the RNN part and the for loop part, and the situation is the same:
def __init__(self, bn=False):
    super(WSDR, self).__init__()
    # Network based on VGG16
    self.conv1 = nn.Sequential(ConvReLU(3, 64, 3, pd=True, bn=bn),
                               ConvReLU(64, 64, 3, pd=True, bn=bn),
                               nn.MaxPool2d(2))
    self.conv2 = nn.Sequential(ConvReLU(64, 128, 3, pd=True, bn=bn),
                               ConvReLU(128, 128, 3, pd=True, bn=bn),
                               nn.MaxPool2d(2))
    self.conv3 = nn.Sequential(ConvReLU(128, 256, 3, pd=True, bn=bn),
                               ConvReLU(256, 256, 3, pd=True, bn=bn),
                               ConvReLU(256, 256, 3, pd=True, bn=bn),
                               nn.MaxPool2d(2))
    self.conv4 = nn.Sequential(ConvReLU(256, 512, 3, pd=True, bn=bn),
                               ConvReLU(512, 512, 3, pd=True, bn=bn),
                               ConvReLU(512, 512, 3, pd=True, bn=bn),
                               nn.MaxPool2d(2))
    self.conv5 = nn.Sequential(ConvReLU(512, 512, 3, pd=True, bn=bn),
                               ConvReLU(512, 512, 3, pd=True, bn=bn),
                               ConvReLU(512, 512, 3, pd=True, bn=bn))

    self.gap = nn.Sequential(ConvReLU(512, 1024, 3, pd=True, bn=bn),
                             ConvReLU(1024, 20, 3, pd=True, bn=bn),
                             nn.AvgPool2d(kernel_size=14, stride=14))
    # adding this fc layer, things go right
    # self.fc = nn.Linear(self.num_classes, self.num_classes)

def forward(self, im_data):
    x = self.conv1(im_data)
    x = self.conv2(x)
    x = self.conv3(x)
    x = self.conv4(x)
    conv5features = self.conv5(x)
    gap = self.gap(conv5features)
    scores = gap.squeeze()
    # scores = self.fc(scores)
    return scores