Investigating this is a little troublesome, since I actually just went on to use another architecture now.
Replicating it with my machine (which runs with CPU by the way), the inputs for the 3 Pads are:
[512, 64, 46, 140] (1,1,1,1)
[512, 32, 92, 280] (1,1,1,1)
[512, 16, 92, 280] (3,3,3,3)

But like I said, it works on my machine. So I can’t determine from here which of the three were the culprit.
Nevertheless, your explanation sounds like it’s more an issue of exploding gradients/values, so input sizes won’t tell you anything anyway, will they?