I use model.train() at test time ,the result is still terrible…

This is definitely problematic.  It means that either your “test” (but really train) data is wrong, or some internal issue with how the loss is calculated.

As I printed out the cls_loss and loc_loss for object detection ,and I realized that my cls_loss is not decreasing …So I think there’re some others issues in my code.

This certainly seems like a key to problem.

I think one of the reason here is the batch normalization layers ,the parameters  momentum  is 0.01,However ,this example only runs 1820 iterations ,I’m not sure except the pre-trained part(frozen),the running mean of bn layers would estimated correctly…

If you run with batchnorm layers in .eval() mode (which your code suggests you are), the running mean and variance are not updated.  You can update only the mean and variance by running in .train() mode and freezing the gradients, which means the affine parameters are not updated.