thejoe:

So, basically I would expect a red area around the out-of-place bolt.


I assume you then have a layer with two outputs that classify your image? I think your resnet output does not need to match locally to the regions of your image.



 thejoe:

Regarding my validation set, I selected similar images belonging to both classes (‘clean’, ‘with bolts’). I tried different shuffling and i obtained different results ( very few cases with low accuracy) but in this way I have a model which is dependent more on the ‘fortune’ of having the right images .


You just overfitted your model. Get more data with more different bolt positions and viewing angles. And use a substantial amount of different images for validation (>1k). As you have synthetic images, I assume you can generate more? How do your true/false pairs look like? Do you use the exactly same scene with and without a bolt? Also are both classes equaly distributed in your training and validation data?