You are much more accurate in your guess than you think - the solution was to use GradCAM! This makes the feature maps repetable and I can now investigate them!
Big thanks to you for your fast and very helpful  insights!