Thank you for your reply,
I am sure that the data processing is the same, and I already shuffle data while training.
I think you make sense of the batch norm problem. I will check the running_mean and running_variance, then try to do something with momentum. However, I don’t think it to have that much of an impact.
This is the result with model.train()
{'rois': array([[448.55865, 435.95306, 648.1225 , 714.6619 ]], dtype=float32),
 'class_ids': array([0], dtype=int64), 
'scores': array([0.9849996], dtype=float32)} 

And this is the result with model.train(False):
{'rois': array([[   0.      ,  921.97406 ,  152.82751 , 1023.      ],
       [ 929.1798  ,  921.1621  , 1023.      , 1023.      ],
       [   0.      ,    0.      ,  159.41663 ,   97.29772 ],
       [ 874.1246  ,    0.      , 1023.      ,   99.22618 ],
       [ 719.88806 ,  820.15857 , 1023.      , 1023.      ],
       [   0.      ,    0.      ,   28.660963,   59.322357],
       [ 991.6581  ,  965.65967 , 1023.      , 1023.      ]],
      dtype=float32),
 'class_ids': array([0, 0, 0, 0, 0, 0, 0], dtype=int64), 
'scores': array([0.22587009, 0.22511227, 0.21677688, 0.2106454 , 0.20411198,
       0.20127359, 0.20020929], dtype=float32)}