Thanks for your kind reply, I finally figure out what goes wrong.
Since my dataset is consists of concatenated patches (shape[35, 3 300, 300]) , when load it with Dataloader, the shape become [batch_size, 35, 3, 300, 300], so I have to tranform it into [batch_size*35, 3, 300, 300], It seems l can’t just flatten the feature tensor, I have to keep the batch dim. Output of my old net  is [C], missing N.
I change my forward code to
def forward(self, x):
        bs = x.size(0) 
        x = self.model.extract_features(x)
        x = x * torch.sigmoid(x)
        x = F.adaptive_avg_pool2d(x, 1).squeeze(-1).squeeze(-1)
        x = x.view(bs, -1)
        x = nn.Linear(35*1536, 1000, bias=True)
        x = nn.ReLU()
        x = nn.Dropout(p=0.5)
        x = nn.Linear(1000, num_classes, bias=True)
        return x

It works, but since it just transforms  tensor from [batch_size * patches, 1536]  to [batch_size, 1536*patches], which seems very ugly, I am wondering  is there any better solutions? Thanks again for your help!