Hey Soumith, yeah I did use eval mode when doing feature extraction.
I just tried the caffe ported version from https://github.com/Cadene/pretrained-models.pytorch 45
and got much better result (close to the baseline reported in the paper). So something might be wrong with the torchvision pretrained resnet