Solved by stoneyang in post #2 
              
              
                The input size should be 299 instead of 224, and there are two losses in the network: a final fc and an intermediate aux_logits. One should notice these difference and adjust the crop_size in data_loader and the computation of metric. Cf. here