Secondly, I tried to remove the dropout layer.
I mean, now the model is like:
class Model(nn.Module):
    def forward(self, input_image):
        out = self.conv_1(input_image) # 20 * 44 * 44
        out = self.pooling_1(out) # 20 * 22 * 22

        out = self.conv_2(out) # 50 * 16 * 16
        out = self.pooling_2(out) # 50 * 8 * 8

        out = self.conv_3(out) # 500 * 2 * 2
        out = self.pooling_3(out) # 500 * 1 * 1

        # out = F.dropout(out) # removed
        out = F.relu(out)
        out = self.conv_4(out) # 2 * 1 * 1

Then the phenomenon of fluctuating has completely disappeared:

validation_acc_no_dropout.png700×450 12 KB


validation_loss_no_dropout.png700×450 9.99 KB

But I don’t know the essential reason, because I have set model.eval()