If you nevertheless get high validation accuracy what is the issue? I am not sure if an activation map can tell you a lot about how the network works? What exactly is your activation map showing?
If you think your networks learned the wrong parts, then maybe you can proprocess the image to only show the region of interest? As Low validation error means your network can classify your validation set very good, are you sure your validation images are correct?