Ok I think I narrowed it down to the basic ResNeXt101_64x4d class 2 since using a different backbone (e.g. densenet) does not produce the slowdown.  I’m guessing the slowdown is in the features 23.  Still bewildered as to why that would cause such a drastic change in performance though.