thanks for your reply. My activation map simply generates an heat map according to weights of the last layer of my resnet50 and then it is overlayed on the image. So, basically I would expect a red area around the out-of-place bolt.
Regarding my validation set, I selected similar images belonging to both classes (‘clean’, ‘with bolts’). I tried different shuffling and i obtained different results ( very few cases with low accuracy) but in this way I have a model which is dependent more on the ‘fortune’ of having the right images .
This is a screenshot of my training performances:

  
      
      ImgBB
  
  
    

Screenshot-2020-04-10-at-11-57-05

Image Screenshot-2020-04-10-at-11-57-05 hosted in ImgBB