title: Contemplating Real-World Object Classification
abstract: Deep object recognition models have been very successful over benchmark
  datasets such as ImageNet. How accurate and robust are they to distribution
  shifts arising from natural and synthetic variations in datasets? Prior research on
  this problem has primarily focused on ImageNet variations (e.g., ImageNetV2,
  ImageNet-A). To avoid potential inherited biases in these studies, we take a
  different approach. Specifically, we reanalyze the ObjectNet dataset recently
  proposed by Barbu et al. containing objects in daily life situations. They showed
  a dramatic performance drop of the state of the art object recognition models on
  this dataset. Due to the importance and implications of their results regarding
  the generalization ability of deep models, we take a second look at their analysis.
  We find that applying deep models to the isolated objects, rather than the entire
  scene as is done in the original paper, results in around 20-30% performance
  improvement. Relative to the numbers reported in Barbu et al., around 10-15%
  of the performance loss is recovered, without any test time data augmentation.
  Despite this gain, however, we conclude that deep models still suffer drastically
  on the ObjectNet dataset. We also investigate the robustness of models against
  synthetic image perturbations such as geometric transformations (e.g., scale,
  rotation, translation), natural image distortions (e.g., impulse noise, blur) as well
  as adversarial attacks (e.g., FGSM and PGD-5). Our results indicate that limiting
  the object area as much as possible (i.e., from the entire image to the bounding
  box to the segmentation mask) leads to consistent improvement in accuracy and
  robustness. Finally, through a qualitative analysis of ObjectNet data, we find that
  i) a large number of images in this dataset are hard to recognize even for humans,
  and ii) easy (hard) samples for models match with easy (hard) samples for humans.
  Overall, our analysis shows that ObjecNet is still a challenging test platform that
  can be used to measure the generalization ability of models. The code and data
  are available in [masked due to blind review].
authors: ali borji
hIndex: ali borji 46